Bug #11960
closedGateway Monitoring Traffic Goes Out Default Gateway
0%
Description
I'm using pfSense Plus 21.02.2 with a SG-3100 and XG-7100 1U. On both systems, I have dual WAN connections with gateway monitoring. I have found that if I lose the non-default gateway WAN, the "monitoring" traffic for that gateway switches over and goes out of the default gateway.
Under normal circumstances, whenever WAN1 is set to monitor 8.8.8.8 and WAN2 is set to monitor 8.8.4.4, the pings for 8.8.8.8 go out of the WAN1 interface and 8.8.4.4 goes out of the WAN2 interface. However, if WAN2 (non-default gateway) goes down, the ping traffic automatically switches to go out of the WAN1 interface. Even after the WAN2 connection is restored (confirmed by the link-state switching to "UP" and the IP address re-populating), the "loss" continues to rise until it reaches 100%. It will STAY at 100% until either the default gateway goes down or dpinger is restarted. If dpinger is restarted, both gateways immediately show as "UP".
I have also tested this in a gateway group set to failover on packet loss or high latency. In this case, if WAN1 is the current gateway and it goes down long enough for WAN2 to become the gateway, then the monitoring traffic for WAN1 will exit the WAN2 interface. Without intervention, WAN1's gateway will remain offline indefinitely. Once dpinger is restarted, both gateways immediately show as back online.
This behavior was not observed in previous versions.
Updated by Jim Pingle about 3 years ago
- Status changed from New to Feedback
- Priority changed from Very High to Normal
This sounds similar to #11296 or another routing issue that was fixed already -- please re-test on a development snapshot and see if you can replicate the behavior there.
Updated by James Blanton about 3 years ago
Jim, Sorry for the delay but I've been out of the office a good bit the past month.
I've updated the SG-3100 to 21.05 and the issue was still there. I just updated to the latest dev. build (21.09.a.20210621.0100) and the bug still exists there as well. I performed a packet capture on my "WAN1" interface and then unplugged my "WAN2" interface. After WAN2's GW was marked as "offline", I began to see the monitoring pings from the WAN2 address going out of the WAN1 interface.
I've also recreated a failover GW group on "packet loss/latency". Anytime the "non-default" gateway goes offline, the traffic is routed out the default gateway instead of the specified gateway.
Updated by James Blanton about 3 years ago
UPDATE! Bug only exists upon "link down"
SETUP:
- Dual WAN connections
- GW group configured as
- failover on "packet loss or high latency"
- WAN1 is tier 1
- WAN2 is tier 2
- GW group is the default route
FINDINGS:
- If you unplug the WAN connection from the router and plug it back in BEFORE pfSense marks it as down, dpinger works as intended
- If you unplug the WAN connection from the router and plug it back in AFTER pfSense marks it as down, dpinger will send the pings for that WAN gateway out of the current/active default gateway and the loss will continue until it reaches 100%, even after WAN connectivity is restored. For example, in the case that WAN1 is the active gateway:
- If WAN2 is unplugged and goes offline, dpinger will route the pings to the monitor IP from the WAN2 IP through WAN1, and WAN2 will continue until loss is 100%
- If WAN1 is unplugged and goes offline, the GW group will switch to WAN2 as the default gateway, and dpinger will route the pings to the monitor IP from the WAN1 IP through WAN2, and WAN1 will continue until loss is 100%
- If the ISP and the WAN interface on the router and connected through a switch, then if ISP is disconnected but the link on the router remains UP, then dpinger works as intended. Even if the gateway goes to 100% loss, as soon as the ISP is reconnected, the loss counter starts going down immediately
In summary, if the gateway goes offline AND the interface link is down, then traffic will be routed out of the default gateway.
Updated by Max Leighton almost 3 years ago
I failed to replicate that in
22.01-DEVELOPMENT (amd64)
built on Fri Nov 05 05:21:41 UTC 2021
FreeBSD 12.3-PRERELEASE
Here were my steps:
1. Start a packet capture on WAN2 listening for ICMP traffic to WAN1's monitor IP.
2. Wait until either 100% packet loss or a lower level of packet loss but still above the failure threshold (tried both)
3. Reconnect WAN1
4. Wait until fail back and packet loss reaches 0%
5. Stop packet capture
When I stop the packet capture, I don't see any ICMP traffic to the WAN1 monitor IP on the WAN2 interface. I tried stopping it before it reached 100% loss and saw the same.
My steps for the WAN2 scenario was as follows:
1. Start packet capture listening on WAN1 for WAN2's monitor IP
2. Disconnect WAN2
3. Wait until either 100% packet loss or a lower level of packet loss but still above the failure threshold
4. Reconnect WAN 2
5. Stop the packet capture
No failover occurs because WAN1 is tier 1, but I still don't see any packets for WAN2's monitor IP on the WAN1 interface.
Updated by Viktor Gurov over 2 years ago
James Blanton wrote in #note-3:
UPDATE! Bug only exists upon "link down"
Unable to reproduce on 22.01.b.20211214.1950 (PPPoE + LTE)
Please provide more detailed information about your configuration, in particular, about the types of network interfaces (PPPoE, DHCP, static, etc.)
Updated by Viktor Gurov over 2 years ago
- Affected Version set to 2.5.2
Updated by Azamat Khakimyanov almost 2 years ago
- Status changed from Feedback to Resolved
Tested on 21.02_2 and on 22.05
I was able to reproduce this issue on 21.02_2 but on 22.05 everything worked correctly:
- when non-default gateway's interface went Down, pfSense stopped sending ICMP requests for non-default gateway's Monitoring IP
- when non-default gateway's interface went UP, pfSense immediately started to send ICMP requests and /Status/Gateway showed that non-default gateway was ONLINE.
On 22.05 there was no issue with default gateway group also.
I marked this Bug as resolved.