Skip to content

Connecting to an IPSec/L2TP VPN with Linux

In my previous post, I described how to set up an IPSec/L2TP server which OS X can connect to out of the box. It seems like IPSec/L2TP has become a popular choice of VPN, given its native support in Windows 7, OS X, iOS and Android. So you think it'd be easy to get Linux connected too, yes? Apparently not. There's a package called l2tp-ipsec-vpn which is supposed to make life easy, but unfortunately it requires Openswan which again requires kernel patches, which I loathe (despite once upon a time patching my kernel several times a day).

All this drove me to write a quick script to connect to an IPSec/L2TP-based VPN using the in-kernel IPSec implementation. The result is here for those who might also find it useful:

Using it is trivial, assuming all goes to plan. And when you need to debug, it prints all the debugging information you might need right to your console. How useful is that?

Setting up IPSec/L2TP for OS X clients

I recently needed to set up a VPN server that OS X would be able to connect to without installing third-party software. The options looked to be IPSec/L2TP or PPTP. With the scary warnings about PPTP being completely broken, I delved into the mildly terrifying world of IPSec/L2TP.

The web is full of pages telling you how to do everything IPSec related with OpenSwan. However, OpenSwan needs out-of-tree kernel modules, which seem like overkill to me (and a support headache), particularly when there's a working implementation already in the Linux kernel. I decided to persevere with the in-kernel NETKEY implementation of IPSec. I've successfully used this implementation of IPSec numerous times before. Yet every single time I do, I have ended up debugging through a process of downloading and reading source code. I can see why many people choose to use OpenSwan! The debugging output from racoon and the in-kernel implementation just isn't particularly informative.

Below is a blow-by-blow account of how I got OS X talking to a IPSec/L2TP using the in-kernel IPSec implementation. Hopefully this will help those who might come here having Googled for the same error messages.

  • I used this excellent guide on the Gentoo wiki, following the sections on ipsec-tools and xl2tpd, and retrofitting for Ubuntu. To avoid repetition, I'm not going to outline those steps again. In short, the Ubuntu-ified packages can be installed using:

    apt-get install ipsec-tools racoon xl2tpd
    This got me most of the way, bar a few stumbling blocks described below. The short summary is also now in the wiki page in the section on Mac OS X.

  • The first error I ran up against seems to be one of the most common from racoon:

    racoon: ERROR: no suitable proposal found.
    racoon: [xx.xx.xx.xx] ERROR: failed to get valid proposal.
    racoon: [xx.xx.xx.xx] ERROR: failed to pre-process ph1 packet (side: 1, status 1).
    racoon: [xx.xx.xx.xx] ERROR: phase1 negotiation failed.

    If you turn up the debugging sufficiently high on racoon (run it from the command-line with racoon -vFdd), you get to see what the desired proposals were. A number of different proposals were sent, but this one was closest match to what I had specified in racoon.conf:

    DEBUG: prop#=1, prot-id=ISAKMP, spi-size=0, #trns=6
    DEBUG: trns#=6, trns-id=IKE
    DEBUG:   lifetime = 3600
    DEBUG:   lifebyte = 0
    DEBUG:   enctype = 3DES-CBC
    DEBUG:   encklen = 0
    DEBUG:   hashtype = SHA
    DEBUG:   authmethod = pre-shared key
    DEBUG:   dh_group = 1024-bit MODP group

    The encryption type, hash type and auth method all match, but that leaves the dh_group. It turns out that the setting of "14" in the instructions refers to the 2048-bit MODP group, whereas here OS X is insisting on using the 1024-bit MODP group.

    Solution: Modify racoon.conf to use "dh_group modp1024".

  • This brings us to the next problem. Phase 1 negotiations begin, but on the server, racoon was showing these errors:

    racoon: NOTIFY: the packet is retransmitted by xx.xx.xx.xx[40276] (1).

    And on the client side (in the Console application on OS X), these messages appear:

    racoon[17952]: IKE Packet: receive failed.  (Initiator, Main-Mode Message 6).

    From the server's point of view, everything is completely normal, but the client retries sending the same packet over and over. After some delving through the source code of racoon on OS X, I tracked the cause to something explained by this comment, near the code for checking the received identifier (ipsec_doi.c:3689):

     * check the following:
     * - In main mode with pre-shared key, only address type can be used.

    Thus, the solution: The my_identifier setting in racoon.conf must be an IP address, not an FQDN as described by the guide. It turns out that this is actually racoon's default, so the simplest solution is just to delete the my_identifier line completely from racoon.conf.

  • The final hurdle was figuring out how to allow clients to connect without hardcoding the IP address of every client (i.e. allow "road-warrior" clients). It turns out that there's an undocumented feature of racoon that allows you to specify a hostname of "*" in psk.txt. For example:

    * mysecretsharedpassword

Huzzah. Several hours of debugging later, a happy outcome. Here are the relevant stanzas from racoon.conf, for reference.

remote anonymous {
        exchange_mode main;

        passive on;
        generate_policy on;
        nat_traversal on;

        proposal_check obey;

        proposal {
                encryption_algorithm 3des;
                hash_algorithm sha1;
                authentication_method pre_shared_key;
                dh_group modp1024;

sainfo anonymous {
        encryption_algorithm aes 256, aes, 3des;
        authentication_algorithm hmac_sha1, hmac_md5;
        compression_algorithm deflate;

6 useful utilities in Mac OS X

If you cared enough to need these, hopefully you would know about them already. Here is a list so that I don't forget again!

  • vmmap - equivalent of /proc/pid/maps
  • fs_usage - print all disk activity caused by a given process, including page in/outs.
  • sc_trace - summary of system calls executed by a process
  • xattr - edit extended attributes of a file (e.g. to kill resource forks).
  • ls -@ - list the extended attribuets of a file
  • mdfind - Spotlight from the command-line.

Firefox on laptops

Here's a neat trick that gives me an extra hour or two of battery life on my MacBook Pro. When I don't need Firefox but would rather not close it, just SIGSTOP it. I'm doing this on Mac OS X, but it almost certainly works on any other *nix system also.

% killall -STOP firefox-bin

And when I want to use it again,

% killall -CONT firefox-bin

Stopping it drops the CPU usage by about 5-10% and causes fewer wakeups. This pushes my battery life from about 4.5 hours to 6.5 hours when I'm just typing away in vim.

Perhaps some day Firefox's idle loop won't be quite so intensive. Though I usually have several Firefox windows open with about 50 tabs in total, so you could argue that I've only myself to blame.


After travelling around Mexico and logging into my mail server from random machines, my paranoia has turned me to looking at one-time passwords. Here's a brief rundown of what I went through to set it up on Ubuntu.

On a local secure machine (laptop/PDA):

  1. apt-get install opie-client
  2. Think up a secret passphrase between 10 and 127 characters long.

On the server:

  1. apt-get install libpam-opie opie-server
  2. As per /usr/share/doc/libpam-opie/README.Debian, I edited /etc/pam.d/ssh and replaced the line
    @include common-auth
    auth	sufficient
    auth 	sufficient
    auth	required
  3. Enable ChallengeResponseAuthentication yes in /etc/ssh/sshd_config.
  4. As the user on the server, run opiepasswd. This will give you a challenge that you need to answer by running opiekey or otp-md5 on a local (secure) machine.
    server% opiepasswd
    Adding bernard:
    You need the response from an OTP generator.
    New secret pass phrase:
            otp-md5 499 so9449
    client% otp-md5 499 so9449
    Using the MD5 algorithm to compute response.
    Reminder: Don't use opiekey from telnet or dial-in sessions.
    Enter secret pass phrase: ***************
    And now pass LIEU FOLK GULL WALL TASK AN back to the opiepasswd command.

Now when you log into the server with ssh, you should be greeted with something like:

insecure% ssh server
otp-md5 498 so9449 ext, Response:
where you respond by running otp-md5 498 so9449 on your local secure machine, entering your secret passphrase, and punch in the response back to the server. If the server rejects it, double-check you didn't mess up your secret passphrase. There are no other warnings or errors if you do!

The sequence number will decrement towards 0, each time a challenge is used. So if you're going travelling without your PDA, you could pre-generate your one-time passwords, print them out and stick them in your wallet (assuming you trust your wallet).

Et voila.