pfSense

Adding this primarily for documentation purposes. Normally with our default ruleset pf will block CARP packets coming in from (self) to prevent L2 packet duplication from affecting the CARP status. However when pfsync is enabled, there is a bit of a quandary here:

• CARP heartbeat packet from primary leaves the primary
• CARP heartbeat packet is delivered both to the secondary and also back to the primary by the misconfigured L2
• The primary rejects this first CARP heartbeat packet and the secondary accepts it (both correct actions)
• The secondary creates a state allowing the heartbeat packet
• The state is synchronized back to the primary
• The primary then accepts subsequent duplicate heartbeat packets, causing itself to be demoted due to the packets arriving faster than it transmits, ending up with flapping VIPs and what appears to the user as a dual backup situation

Disabling pfsync and clearing the CARP states allows the CARP VIPs to function until the L2 is repaired.

Not sure if there is any good way around that. Given that it's an L2 problem there is only so much we can do to stop foot-shooting in that area. Perhaps pass carp in expected directions/interfaces with no state?