pfSense

from what we discussed on this previously, rather than selecting a default or not, there should be 3 options in a drop down - yes (it's the default unless it's down), no (it's never the default), and eligible (it can become the default if the default is down).

I'm not sure this behavior of changing gateways should be retained at all for v2.0.

One of the issues that came up was with Racoon. Even if the default gateway came back to the WAN interface the tunnels never came back up. Restarting racoon did not prove to fix things. I ended up rebooting the firewall before everything came back up normally. The WAN interface was never actually down in my case, but the monitor IP failed for roughly 30 seconds which was enough to cause the default switch.

What may have happened here is that there might have been GRE traffic states created on the OPT interface which could then cause issues. IPsec communicates over UDP for the control but GRE is used for the transport. The few tunnels that were bound to the OPT interface kept working, also after a restart. Leading me to believe it might be a state issue or assymetric routing.

Static IPsec tunnels on the OPT interface employ static routes to make sure that all traffic goes out the correct gateway. In my case Racoon was also bound to a CARP vip as this is a Carp cluster and this is the main VPN server.

Gateway failover does not make much sense for a CARP cluster really, all tunnels are terminated via the WAN interface.

Looks like this is related to ticket #1508. I never checked that the route actually failed back.

Seth's referring to ESP rather than GRE there

The short version is racoon need to learn to differentiate packets by their incoming interface.
A patch is present on racoon mailing lists but also a RECVIF sockopt might help in this case as well.