Routing fail on CARP IPsec
There is some strange detail in pfSense, found on 2.2.6 and 2.2.2 versions.
Probably may affect all versions, just didn't tested on the rest of them yet.
Trying to describe an environment and conditions:
I have 2 pfSense instances, running in HighAvail-mode with CARP on virtual interfaces on ESXi 5.5.0.
All set according to documentation, and failsafe mode is functioning just fine.
There is a IPsec tunnel between this cluster of pfS's and FortiGate200B.
The tunnel is also functioning just fine, except of one little detail, which is the next:
I'm testing the HighAvailability mode by pinging a host located inside internal LAN,
from a machine located in another internal LAN, connected through the IPsec tunnel.
During a pingin' process I'm shutting down a primary instance of pfSense, to check the work of HighAvailability.
The secondary pfSense instance is becoming Master instead of Backup on CARP interfaces and tunnel is still
BUT when I'm turning on a primary pfSense instance and it takes the Master state back on itself,
the ping returns DHU (Destination Host Unreachable), and a icmp packet comes back from an address of the real LAN interface,
of primary pfSense.
The gateway on the pinging machine is configured through the Virtual CARP LAN IP.
The log is looking like this:
knoppix@Microknoppix:~$ ping Remote_host_IP PING Remote_host_IP (Remote_host_IP) 56(84) bytes of data. 64 bytes from Remote_host_IP: icmp_req=1 ttl=126 time=156 ms ... 64 bytes from Remote_host_IP: icmp_req=20 ttl=126 time=156 ms 64 bytes from Remote_host_IP: icmp_req=21 ttl=126 time=155 ms //here the primary pfSense goes down and being replaced by a secondary CARP member, //that takes about 20 seconds, so I get about 20 packets with timeout. 64 bytes from Remote_host_IP: icmp_req=40 ttl=126 time=156 ms 64 bytes from Remote_host_IP: icmp_req=41 ttl=126 time=155 ms ... 64 bytes from Remote_host_IP: icmp_req=73 ttl=126 time=155 ms 64 bytes from Remote_host_IP: icmp_req=74 ttl=126 time=155 ms //and here the primary pfSence takes the master state back From pfSense_LAN_IP icmp_seq=83 Destination Host Unreachable From pfSense_LAN_IP icmp_seq=84 Destination Host Unreachable From pfSense_LAN_IP icmp_seq=85 Destination Host Unreachable From pfSense_LAN_IP icmp_seq=86 Destination Host Unreachable From pfSense_LAN_IP icmp_seq=87 Destination Host Unreachable ^C
and so on.
When I start a new ping session, everything is okay again.
The bug is happening only when a pfSence with a lower advskew takes the Master role.
Tried to solved by myself, but failed. Maybe you will succeed.