pfSense

Jim, Sorry for the delay but I've been out of the office a good bit the past month.

I've updated the SG-3100 to 21.05 and the issue was still there. I just updated to the latest dev. build (21.09.a.20210621.0100) and the bug still exists there as well. I performed a packet capture on my "WAN1" interface and then unplugged my "WAN2" interface. After WAN2's GW was marked as "offline", I began to see the monitoring pings from the WAN2 address going out of the WAN1 interface.

I've also recreated a failover GW group on "packet loss/latency". Anytime the "non-default" gateway goes offline, the traffic is routed out the default gateway instead of the specified gateway.

UPDATE! Bug only exists upon "link down"

SETUP:

1. Dual WAN connections
2. GW group configured as
   1. failover on "packet loss or high latency"
   2. WAN1 is tier 1
   3. WAN2 is tier 2
3. GW group is the default route

FINDINGS:

1. If you unplug the WAN connection from the router and plug it back in BEFORE pfSense marks it as down, dpinger works as intended
2. If you unplug the WAN connection from the router and plug it back in AFTER pfSense marks it as down, dpinger will send the pings for that WAN gateway out of the current/active default gateway and the loss will continue until it reaches 100%, even after WAN connectivity is restored. For example, in the case that WAN1 is the active gateway:
   1. If WAN2 is unplugged and goes offline, dpinger will route the pings to the monitor IP from the WAN2 IP through WAN1, and WAN2 will continue until loss is 100%
   2. If WAN1 is unplugged and goes offline, the GW group will switch to WAN2 as the default gateway, and dpinger will route the pings to the monitor IP from the WAN1 IP through WAN2, and WAN1 will continue until loss is 100%
3. If the ISP and the WAN interface on the router and connected through a switch, then if ISP is disconnected but the link on the router remains UP, then dpinger works as intended. Even if the gateway goes to 100% loss, as soon as the ISP is reconnected, the loss counter starts going down immediately

In summary, if the gateway goes offline AND the interface link is down, then traffic will be routed out of the default gateway.

I failed to replicate that in

22.01-DEVELOPMENT (amd64)
built on Fri Nov 05 05:21:41 UTC 2021
FreeBSD 12.3-PRERELEASE

Here were my steps:

1. Start a packet capture on WAN2 listening for ICMP traffic to WAN1's monitor IP.
2. Wait until either 100% packet loss or a lower level of packet loss but still above the failure threshold (tried both)
3. Reconnect WAN1
4. Wait until fail back and packet loss reaches 0%
5. Stop packet capture

When I stop the packet capture, I don't see any ICMP traffic to the WAN1 monitor IP on the WAN2 interface. I tried stopping it before it reached 100% loss and saw the same.

My steps for the WAN2 scenario was as follows:

1. Start packet capture listening on WAN1 for WAN2's monitor IP
2. Disconnect WAN2
3. Wait until either 100% packet loss or a lower level of packet loss but still above the failure threshold
4. Reconnect WAN 2
5. Stop the packet capture

No failover occurs because WAN1 is tier 1, but I still don't see any packets for WAN2's monitor IP on the WAN1 interface.

Looks like a duplicate of #11570

James Blanton wrote in #note-3:

UPDATE! Bug only exists upon "link down"

Unable to reproduce on 22.01.b.20211214.1950 (PPPoE + LTE)

Please provide more detailed information about your configuration, in particular, about the types of network interfaces (PPPoE, DHCP, static, etc.)