Bug #9658
closed
Gateway monitor IPs are being put into the routing table
0%
Description
As the subject says, fpsense puts the IP addresses that are configured as monitor IPs for gateways in the routing table.
This leads to traffic to those monitor IPs being sent out to the wrong interface which causes different issues:
Let's look at the behavior with the below pretty-much-standard configuration:
- two WAN interfaces
- the gateways of those WAN interfaces configured with monitor IPs 1.1.1.1 and 8.8.8.8
- a gateway group configured for failover
- that gateway group set as default gateway
- WAN1 interface with monitor IP 1.1.1.1 is the primary (Tier 1) internet connection
- WAN2 interface with monitor IP 8.8.8.8 is the secondary (Tier 2) internet connection
Error scenario 1: Everything is running normally, i.e. both WAN interfaces are up, the monitor IPs are both reachable and WAN1 is active as configured via the Gateway Group Tier.
Although WAN1 is the primary and active WAN interface, all traffic destined to 8.8.8.8 (be it from pfsense itself in case that nameserver is configured somewhere, or be it from clients behind the firewall that may use that DNS server) will actually go through WAN2. Many people probably don't even notice this, yet, it is not correct behaviour, as the firewall is configured to use the primary WAN1 interface for all outgoing traffic as long as WAN1 is not down.
Error scenario 2: WAN1 goes down (or, to be precise, the monitor IP 1.1.1.1 does not respond to pings anymore)
Pfsense will detect that 1.1.1.1 is not pingable anymore and switch over to WAN2. But the route to 1.1.1.1 is still there and thus, 1.1.1.1 becomes completely unreachable for both the firewall as well as the clients behind it. Which is certainly not correct behaviour.
Error scenario 3: WAN2 goes down (or, to be precise, the monitor IP 8.8.8.8 does not respond to pings anymore)
Pfsense will detect that 8.8.8.8 is not pingable, but since WAN2 is the secondary (Tier 2) WAN interface, won't switch over or do anything. But the route to 8.8.8.8 is still there and thus 8.8.8.8 becomes completely unreachable for both the firewall as well as the clients behind it, although the primary WAN1 internet connection is still working fine. Which is certainly not correct behaviour.
How to fix: Simply don't put those IPs in the routing table. Dpinger already binds to the corresponding interface IP address and thus, that monitor traffic is being sent out via the correct interface. There is no need for those routes, all they do is cause the above issues.
Updated by Jim Pingle almost 6 years ago
- Status changed from New to Not a Bug
- Priority changed from High to Normal
This is by design. It has to be that way, or it can't be sure that the monitor address will ping via the correct interface.
Even when you bind to an interface, it still respects the routing table.
Updated by Anonymous almost 6 years ago
In scenario 1, the firewall is sending traffic out to interfaces it's not supposed to do.
In scenario 2, the monitor IP becomes completely unreachable although it's reachable just fine over the other WAN interface
In scenario 3, the monitor IP becomes completely unreachable, just because the currently inactive WAN connection goes down.
All these things are not documented anywhere and are certainly not how one expects a firewall/router to behave. It may be that it's designed that way and "not a bug" by your definition, but still, it is broken. In fact, I had to abort a firewall migration to Pfsense because I was bitten by scenario 3, did not have one of the two WAN connections connected and could not reach DNS because of this.
Even when you bind to an interface, it still respects the routing table.
I have actually tested that and cannot reproduce that behaviour, i.e. when I simply delete the routes, dpinger still uses the correct interfaces.
This is the routing table with no change, routes to 1.1.1.1 and 8.8.8.8 are there:
Internet:
Destination Gateway Flags Netif Expire
default xx.xx.xx.57 UGS igb2
1.1.1.1 xx.xx.xx.145 UGHS igb0
8.8.8.8 xx.xx.xx.57 UGHS igb2
xx.xx.xx.144/29 link#1 U igb0
xx.xx.xx.147 link#1 UHS lo0
127.0.0.1 link#7 UH lo0
172.16.0.0/16 link#2 U igb1
172.16.1.124 link#2 UHS lo0
192.168.1.0/24 link#4 U igb3
192.168.1.1 link#4 UHS lo0
xx.xx.xx.56/29 link#3 U igb2
xx.xx.xx.59 link#3 UHS lo0
ICMP packets are being sent out as expected:
running tcpdump -n -i igb0 "host 1.1.1.1" shows pings going out to the correct interface igb0:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:31:12.291392 IP xx.xx.xx.147 > 1.1.1.1: ICMP echo request, id 7765, seq 387, length 8
08:31:12.304111 IP 1.1.1.1 > xx.xx.xx..147: ICMP echo reply, id 7765, seq 387, length 8
08:31:12.823635 IP xx.xx.xx..147 > 1.1.1.1: ICMP echo request, id 7765, seq 388, length 8
08:31:12.836103 IP 1.1.1.1 > xx.xx.xx..147: ICMP echo reply, id 7765, seq 388, length 8
running tcpdump -n -i igb2 "host 1.1.1.1" shows pings are NOT going out over the other WAN interface:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
running tcpdump -n -i igb2 "host 8.8.8.8" shows pings going out to the correct interface igb2:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2, link-type EN10MB (Ethernet), capture size 262144 bytes
08:34:11.931160 IP xx.xx.xx.59 > 8.8.8.8: ICMP echo request, id 8071, seq 727, length 8
08:34:11.935138 IP 8.8.8.8 > xx.xx.xx..59: ICMP echo reply, id 8071, seq 727, length 8
08:34:12.463423 IP xx.xx.xx..59 > 8.8.8.8: ICMP echo request, id 8071, seq 728, length 8
08:34:12.467359 IP 8.8.8.8 > xx.xx.xx..59: ICMP echo reply, id 8071, seq 728, length 8
running tcpdump -n -i igb0 "host 8.8.8.8" shows pings are NOT going out over the other WAN interface:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
Now delete those routes:
route delete 1.1.1.1
route delete 8.8.8.8
As expected, the routing table now looks like this:
Destination Gateway Flags Netif Expire
default xx.xx.xx.57 UGS igb2
xx.xx.xx.144/29 link#1 U igb0
xx.xx.xx.147 link#1 UHS lo0
127.0.0.1 link#7 UH lo0
172.16.0.0/16 link#2 U igb1
172.16.1.124 link#2 UHS lo0
192.168.1.0/24 link#4 U igb3
192.168.1.1 link#4 UHS lo0
xx.xx.xx.56/29 link#3 U igb2
xx.xx.xx.59 link#3 UHS lo0
tcpdump -n -i igb0 "host 1.1.1.1" shows pings going out to the correct interface igb0:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:39:39.701855 IP xx.xx.xx.147 > 1.1.1.1: ICMP echo request, id 7765, seq 1345, length 8
08:39:39.714254 IP 1.1.1.1 > xx.xx.xx.147: ICMP echo reply, id 7765, seq 1345, length 8
08:39:40.233110 IP xx.xx.xx.147 > 1.1.1.1: ICMP echo request, id 7765, seq 1346, length 8
08:39:40.245173 IP 1.1.1.1 > xx.xx.xx.147: ICMP echo reply, id 7765, seq 1346, length 8
running tcpdump -n -i igb2 "host 1.1.1.1" shows pings are NOT going out over the other WAN interface, as expected:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
tcpdump -n -i igb2 "host 8.8.8.8" shows pings going out to the correct interface igb2:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2, link-type EN10MB (Ethernet), capture size 262144 bytes
08:42:39.600816 IP xx.xx.xx.59 > 8.8.8.8: ICMP echo request, id 8071, seq 1685, length 8
08:42:39.604761 IP 8.8.8.8 > xx.xx.xx.59: ICMP echo reply, id 8071, seq 1685, length 8
08:42:40.128565 IP xx.xx.xx.59 > 8.8.8.8: ICMP echo request, id 8071, seq 1686, length 8
08:42:40.132520 IP 8.8.8.8 > xx.xx.xx.59: ICMP echo reply, id 8071, seq 1686, length 8
tcpdump -n -i igb0 "host 8.8.8.8" shows pings are NOT going out over the other WAN interface, as expected:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
So, to me it looks like the solution is to delete those routes. Atleast I cannot find any ill effect when doing that, in fact, everything starts working as it should, i.e. the monitor IPs stay reachable for clients in scenarios 1 to 3 and WAN failover works fine.