pfSense

We have approximately 30 ipsec tunnels on a netgate SG-3100. We've been getting errors that stop tunnels from coming up and require a reboot to recover.

Found that we're getting this kind of error message at the time of the problems:

charon: 12[KNL] <con13000|200> unable to add SAD entry with SPI cb36cc85: Cannot allocate memory (12)

turned on ipsec debugging and find this message when the memory problem happens:

kernel: key_setsaval: unable to initialize SA type 3.

I've reported this bug to freebsd as https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247163
My investigation of the ipsec code appears to have found that cesa is limited to 64 active sessions at a time and will return enomem when it runs out of sessions.
All 11 releases of freebsd appear to have this limit. The 12 releases appear to have different memory management and do not have this problem.

we have changed net.inet.ipsec.crypto_support to the software driver only value of 33554432 and are no longer seeing problems with vpns.

discussion on the problem in the forum:
https://forum.netgate.com/topic/153791/vpns-disconnecting-reported-memory-issue/8