Setting up an IPSec/L2TP VPN Server

OS X integration at no extra cost!

Setting up a VPN server is one of the basic parts of setting up my personal servers. It’s a useful little thing to have, whether it’s for travelling abroad and wanting to view my US Netflix library, or for more serious usage, like protecting your traffic on an untrusted network. Either way, I like to have one set up. Usually, OpenVPN is the go-to server to use, and there are a million and one guides out there that tell you how to set it up. However, I was interested in using something that didn’t require a separate client installation on just about every platform. That narrows down the options quite a bit. For this, I decided to make an L2TP VPN server. Windows and OS X both have integrated clients, which makes it easy to set up on the client end. The same isn’t true on the server end, however.

Compared to OpenVPN, setting up L2TP is far more involved. There are a few different options, and for a time, I used Softether because it was stupidly simple to set up. Softether’s greatest advantage is that it does everything: OpenVPN, L2TP, SSTP, and its own custom protocol. One server is capable of accepting connections from just about every major VPN client out there. Amazing, right? Sure, but there are a few major annoyances with SoftEther.

  1. Package maintenance is abysmal. SoftEther doesn’t release packages for any specific distro, and none of the major distros make packages for it. It’s distributed as a compressed tarball that contains the software, which brings me to my next issue…
  2. It’s entirely self-contained. This is similar to proprietary software packages like Google Chrome or Steam for Linux, but there’s no good reason for an open source project to be distributed with its own versions of libraries and sch. Sure, it makes it far more portable in the sense that there are no dependencies to worry about, but you’re stuck using the (possibly insecure) versions of the packages they built the distribution with.
  3. Configuration is a nightmare. SoftEther’s configuration isn’t done through text files, but rather through their config utility. You use it to set up your VPN networks and edit the settings for the server, as well as for the client if you use the Linux/OS X SoftEther client. This is a command-line only tool with poor documentation, and the only GUI version is for Windows.
  4. No initscripts. Only the config utility, which you use to launch the daemons as well.

In essence, SoftEther is a pain to maintain (no package support means no automatic updating!) and to use in general. One could argue that configuration and such need only be done once, but still. I’d prefer that the configuration make sense and be something in /etc that I can easily move about between distributions and software upgrades. So, I set about looking into how to do it myself.


For IPSec, you’ve got a few options. There’s ipsec-tools, which is a port of the BSD IPSec tools to Linux, and there’s FreeS/WAN and all its derivatives. I personally chose ipsec-tools simply because it was the more familiar one (I’ve seen racoon a lot during my time troubleshooting OS X’s VPN tools). Following the guide at the Gentoo Wiki, I got started. First, you ought to install ipsec-tools.

$ emerge -av ipsec-tools

This will install racoon, which is the server we’ll be using for IPSec. In essence, this provides the transport security for our VPN connection. Once racoon is installed, we need to set it up. For my purposes, I went with a shared key setup. This is simpler to do and more than enough for my single-person server. The first file is the config file, /etc/racoon/racoon.conf.

path pre_shared_key "/etc/racoon/psk.txt";

remote anonymous {
    exchange_mode main;
    my_identifier address ""; # insert your server's IP address here

    passive on;
    generate_policy on;
    nat_traversal on;

    # for windows
    proposal {
        encryption_algorithm aes;
        hash_algorithm sha1;
        authentication_method pre_shared_key;
        dh_group modp2048;

    # for os x
    proposal {
        encryption_algorithm aes;
        hash_algorithm sha1;
        authentication_method pre_shared_key;
        dh_group modp1024;

sainfo anonymous {
    encryption_algorithm aes 256, rijndael 256, blowfish 448;
    authentication_algorithm hmac_sha1;
    compression_algorithm deflate;

A few things to note here. This config will use the PSK file at /etc/racoon/psk.txt. Gentoo creates it automatically, you should put it there and make it only readable/writable by root if it’s not there. The identifier must be an address for OS X clients. FQDN doesn’t work. The encryption algorithms and hashing algorithms are the best options that OS X supports. The guide generally suggest 3DES + MD5, but those seemed like poor options to me. I’d go for SHA256/SHA512 if I could, but OS X would throw up for anything better than SHA1. Same with the DH group. It only like modp1024.

With this in place, you should set up your PSK. Unless you know the IP address of your client beforehand (which most people probably don’t), you should use a catch-all like I do.

# /etc/racoon/psk.txt

The general format for the file is an IP address-PSK pair on each line, and the asterisk here is a wildcard. Next, set up the policy. This might not be necessary for everyone (the guide insinuates it shouldn’t be), but I was getting errors with the L2TP daemon if I didn’t do this. You should put this in /etc/ipsec-tools.conf.

#!/usr/sbin/setkey -f

spdadd[l2tp] udp -P out ipsec esp/transport//require;
spdadd[l2tp] udp -P in ipsec esp/transport//require;

Be sure to put in your server’s IP address in place of After that, you should set up IPtables to allow IPSec traffic through.

$ iptables -A INPUT -p esp -j ACCEPT
$ iptables -A INPUT -p udp --dport 500 -j ACCEPT
$ iptables -A INPUT -p udp --dport 4500 -j ACCEPT

With that done, IPSec should be just about done. Go ahead and start up racoon and add it to the default runlevel.

$ /etc/init.d/racoon start
$ rc-update add racoon default


The L2TP configuration is simpler. The guide listed both xl2tpd and rp-l2tp as options. I went with xl2tpd because it included the ability to do DHCP for you. First, install it.

$ emerge -av xl2tpd

Once that’s done, go ahead and set up the config.

port = 1701
access control = no

[lns default]
ip range =
local ip =
require authentication = yes
name = MyVPN
length bit = yes
pppoptfile = /etc/ppp/options.xl2tpd

You can change the name/IP address stuff if you’d like. This is the basic config, really, with DHCP automatically enabled as part of the daemon. It’ll assign addresses in the range, with the VPN server at Next, set up the options. I used the default ones. This goes into /etc/ppp/options.xl2tpd.

mtu 1410
mru 1410

The last config file is the authentication one. This is at /etc/ppp/chap-secrets.

# client    server  secret                IP addresses
USERNAME    *       PASSWORD_HERE         *

Substitute in the username and password you want to use. You can add as many clients as you’d like.

This file will store the authentication data in cleartext. Make sure it’s only viewable/writable by root.

With that clarified, you now have all you need for a working L2TP server. However, we want to make sure it only allows connections from IPSec tunnels, since L2TP is hilariously insecure on its own.

$ iptables -t filter -A INPUT -p udp -m policy --dir in --pol ipsec -m udp --dport l2tp -j ACCEPT
$ iptables -t filter -A INPUT -p udp -m udp --dport l2tp -j REJECT --reject-with icmp-port-unreachable
$ iptables -t filter -A OUTPUT -p udp -m policy --dir out --pol ipsec -m udp --sport l2tp -j ACCEPT
$ iptables -t filter -A OUTPUT -p udp -m udp --sport l2tp -j REJECT --reject-with icmp-port-unreachable

With that set up, your server ought to be working and able to accept connections. Go ahead and add xl2tp to the default runlevel and fire it up.

$ /etc/init.d/xl2tpd start
$ rc-update add xl2tpd default

Then, go ahead and try to connect to the L2TP server with your information. The connection should work, though your computer won’t be able to reach the internet. You’ve got to configure your server to forward the packets. For me, it was a matter of iptables rules and setting some settings in sysctl.conf.

$ iptables -A INPUT -i ppp0 -j ACCEPT
$ iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
$ iptables -A FORWARD -i ppp0 -j ACCEPT

This is assuming that ppp0 will be the interface used by the server. Adjust as necessary.

# /etc/sysctl.conf
net.ipv4.ip_forward = 1

Windows Clients

Windows clients require a fairly specific settings combo to work. Go to Network Connections, right click the VPN connection, and click Properties. Under the Security tab, make sure the Type is set to ‘Layer 2 Tunneling Protocol with IPsec (L2TP/IPsec)’, then in Advanced Settings set your PSK. For Data encryption, use ‘Maximum strength encryption’ (any of the other options will cause the IPsec negotiation to fail). For Authentication, set to the second option (Allow these protocols) and ensure CHAP is selected (not MS-CHAP v2).