This weekend while setting up RRAS as part of a blog post I am working on a blog posting for Cloudservers.com, I had an issue where an L2TP VPN worked from my Windows 8 PC, but not from my Mac. The GUI error was saying that the VPN was inaccessible, and after looking further in Console, I saw this:

raccoon: not acceptable Identity Protection Mode

The resolution was to edit the /etc/racoon/racoon.conf file. To do this, fire up Terminal or any app (I use iTerm2 it’s awesome), switch to su. Enter your password, and then type in “nano /etc/racoon/racoon.conf” and swap the exchange mode setting from aggressive,main to main,aggressive. Both of these entires were in my file, so I just commented out the second entry:

My connection to my Windows Server 2012 L2TP VPN from Mac OSX now works as expected!

Connect to L2TP VPN on Mac OSX
  • K Man

    Thanks for this tip. It works for me. The only thing is that the settings revert to the old setting after a reboot. Do you have any idea how to make it permanent?

  • K Man

    It turns out that changing the exhange_mode and restarting VPN isn’t the solution.

    It’s enough to just restart VPN to fix the problem.

    This thread explains the real problem and a solution:

    https://discussions.apple.com/thread/6880474

    • Gabe

      Interesting. I’ve not had to use this type of VPN on my Mac in over a year and when I did it was for a limited time. Glad you found a fix!

  • Lord_John_Whorfin

    Thank you so much! Ever since I upgraded to 10.11.3 the VPN has been totally flaky, working only once after I started it and then rejecting connections from my iPhone and iPad. I made the change you suggested and it’s now working great. It’s so cool that a 3 year old post is saving my bacon! Thanks!

%d bloggers like this: