Bernard's Babble

I recently needed to set up a VPN server that OS X would be able to connect to without installing third-party software. The options looked to be IPSec/L2TP or PPTP. With the scary warnings about PPTP being completely broken, I delved into the mildly terrifying world of IPSec/L2TP.

The web is full of pages telling you how to do everything IPSec related with OpenSwan. However, OpenSwan needs out-of-tree kernel modules, which seem like overkill to me (and a support headache), particularly when there's a working implementation already in the Linux kernel. I decided to persevere with the in-kernel NETKEY implementation of IPSec. I've successfully used this implementation of IPSec numerous times before. Yet every single time I do, I have ended up debugging through a process of downloading and reading source code. I can see why many people choose to use OpenSwan! The debugging output from racoon and the in-kernel implementation just isn't particularly informative.

Below is a blow-by-blow account of how I got OS X talking to a IPSec/L2TP using the in-kernel IPSec implementation. Hopefully this will help those who might come here having Googled for the same error messages.

• I used this excellent guide on the Gentoo wiki, following the sections on ipsec-tools and xl2tpd, and retrofitting for Ubuntu. To avoid repetition, I'm not going to outline those steps again. In short, the Ubuntu-ified packages can be installed using:

  apt-get install ipsec-tools racoon xl2tpd
  

  This got me most of the way, bar a few stumbling blocks described below. The short summary is also now in the wiki page in the section on Mac OS X.

• The first error I ran up against seems to be one of the most common from racoon:

  racoon: ERROR: no suitable proposal found.
  racoon: [xx.xx.xx.xx] ERROR: failed to get valid proposal.
  racoon: [xx.xx.xx.xx] ERROR: failed to pre-process ph1 packet (side: 1, status 1).
  racoon: [xx.xx.xx.xx] ERROR: phase1 negotiation failed.
  

  If you turn up the debugging sufficiently high on racoon (run it from the command-line with racoon -vFdd), you get to see what the desired proposals were. A number of different proposals were sent, but this one was closest match to what I had specified in racoon.conf:

  DEBUG: prop#=1, prot-id=ISAKMP, spi-size=0, #trns=6
  DEBUG: trns#=6, trns-id=IKE
  DEBUG:   lifetime = 3600
  DEBUG:   lifebyte = 0
  DEBUG:   enctype = 3DES-CBC
  DEBUG:   encklen = 0
  DEBUG:   hashtype = SHA
  DEBUG:   authmethod = pre-shared key
  DEBUG:   dh_group = 1024-bit MODP group
  

  The encryption type, hash type and auth method all match, but that leaves the dh_group. It turns out that the setting of "14" in the instructions refers to the 2048-bit MODP group, whereas here OS X is insisting on using the 1024-bit MODP group.

  Solution: Modify racoon.conf to use "dh_group modp1024".

• This brings us to the next problem. Phase 1 negotiations begin, but on the server, racoon was showing these errors:

  racoon: NOTIFY: the packet is retransmitted by xx.xx.xx.xx[40276] (1).
  

  And on the client side (in the Console application on OS X), these messages appear:

  racoon[17952]: IKE Packet: receive failed.  (Initiator, Main-Mode Message 6).
  

  From the server's point of view, everything is completely normal, but the client retries sending the same packet over and over. After some delving through the source code of racoon on OS X, I tracked the cause to something explained by this comment, near the code for checking the received identifier (ipsec_doi.c:3689):

  /*
   * check the following:
   * - In main mode with pre-shared key, only address type can be used.
  ...
  

  Thus, the solution: The my_identifier setting in racoon.conf must be an IP address, not an FQDN as described by the guide. It turns out that this is actually racoon's default, so the simplest solution is just to delete the my_identifier line completely from racoon.conf.

• The final hurdle was figuring out how to allow clients to connect without hardcoding the IP address of every client (i.e. allow "road-warrior" clients). It turns out that there's an undocumented feature of racoon that allows you to specify a hostname of "*" in psk.txt. For example:

  * mysecretsharedpassword
  

Huzzah. Several hours of debugging later, a happy outcome. Here are the relevant stanzas from racoon.conf, for reference.

remote anonymous {
        exchange_mode main;

        passive on;
        generate_policy on;
        nat_traversal on;

        proposal_check obey;

        proposal {
                encryption_algorithm 3des;
                hash_algorithm sha1;
                authentication_method pre_shared_key;
                dh_group modp1024;
        }
}

sainfo anonymous {
        encryption_algorithm aes 256, aes, 3des;
        authentication_algorithm hmac_sha1, hmac_md5;
        compression_algorithm deflate;
}