pfSense

I have a very large pfsense installation with over 32 vpn connections. The HA pair's config was originally built with (I think) 4.2.x with both hardware and software upgraded over the years so all of the VPN's are tunnels. I have been in the process of converting them from tunneled to VTI.

I recently discovered a MAJOR issue with the way pfsense is assigning VTI interfaces. the reqid of the VTI interface is being assigned a value of the IKEID * 1000. However reqid appears to be only a 16 bit signed integer as any ipsec vti tunnel with an IKEID over 32 fails. IE a ikeid of 33 yields a reqid of 33000 and the interface errors out and does not come up.

Unfortunately it has been 3 weeks since I discovered the issue and my log data of the exact error message generated in the system log has been cleared.

To reproduce this error simply create a IPSEC phase1 with a VTI phase 2 between 2 pfsense systems. One one end save just the IPSEC configuration. Assuming the ipsec is ikeid 1 (con1000 etc) find and replace all in the xml '<ikeid>1</ikeid>' with '<ikeid>33</ikeid>' and restore the modified configuration. Reboot the pfsense and the IPSEC tunnel will not work.

I did a similar thing to fix my effected tunnels after I deleted some old inactive tunnels to get my effected VTI tunnels with an ikeid over 33 under 33 so they would work.

32 tunnels I know is a lot of tunnels but in my case I will need many more. I have not heard of anyone using pfsense even close to the scale we are using it for.

My HA pair hardware:
Intel(R) Xeon(R) CPU E5-2470 0 @ 2.30GHz
16 CPUs: 2 package(s) x 8 core(s)
AES-NI CPU Crypto: Yes (active)
32GB Ram

A background on our application case we currently have a 2 server HA pair 20 vlan interfaces with carp on a two port LAGG with x2 10gb Intel interfaces assigned. It currently has 25 IPSEC VTI tunnels, 10 IPSEC Legacy tunnels and 3 Openvpn servers. We use them to route private traffic between 300 servers, multiple satellite data centers and 34 offices in 8 countries. All of which are PFSense installations. Of course the endpoints are not nearly as powerful servers.

While I do intend on separating some of the traffic to other auxiliary pfsense servers we have at the data center for the moment this is not possible until I have all of our IPSEC VPN's converted to VTI. With the current limitations I have a bit of a chicken/egg problem if I cannot have more than 32 IPSEC interfaces at a time.

Our pfsense servers run at less than 25% cpu usage so I have plenty of headroom for more when this issue is resolved.