Updated 12/10/2013: Jon Stacey has identified a fix, which involves restoring a copy of racoon from Mountain Lion. I followed his advice and was able to successfully establish an L2TP VPN connection again. Please visit the ‘OS X 10.9 Mavericks fix’ section of his page for more information.
Updated 07/06/2014: The VPN continues to function just fine. I’ve installed several server updates since my previous updates so at this point I’m no longer certain if the previous fix is still in place or if Apple provided a proper fix via update. Regardless, I haven’t had any new problems.
Despite having updated various software packages to avoid software problems with compatibility under Mavericks I hit a wall that I simply haven’t been able to get past. The built-in VPN server in Mavericks (with and without OS X Server) is broken.
I had it working just fine under Mountain Lion, which I managed using iVPN. After upgrading it stopped working. Not long after I added OS X Server and it still wouldn’t work (I purchased OS X Server for its Time Machine capabilities, not for VPN support so I didn’t purchase it expecting it to fix this problem).
Prior to upgrading I had an L2TP VPN server working. Since upgrading I’ve been unable to get L2TP or PPTP working, though I’ve seen more activity with PPTP during connection attempts than with L2TP.
I have hunted across many forum threads and I have been unable to find a solution that works. Some users have reported success though the apparent causes and solutions are extremely varied. The majority of posts are from users that cannot find a solution. This weekend I spent several hours trying to find a solution without success.
At this point there appears to be nothing more that I can do. I’ll simply have to wait until Apple issues an update that fixes the problem. Until then I’ll have to use a service such as LogMeIn to access my desktop, though it’s only a partial solution and doesn’t provide the full access to my network that I need from a VPN server.