High Availability Setup with Gateway to secondary pfSense not working - No Internet
the following setup is working just fine on pfSense CE 2.6.0:
- High Availability/CARP
- Gateway group WAN_GROUP consisting of:
- WAN_PPPOE Tier 1
- VLAN002_GATEWAY Tier 5
- Trigger level: Member down
- VLAN002_GATEWAY points to CARP VIP in VLAN 2
- VLAN 2:
- pfsense 10.0.2.1 (CARP VIP)
- pfSense-a 10.0.2.2
- pfSense-b 10.0.2.3
- Default IPv4 Gateway: WAN_GROUP
- Default IPv6 Gateway: WAN_DHCP6
This should have the following effect:
If pfsense-a is CARP MASTER and has PPPOE active, then the default ipv4 gateway is WAN_PPPOE.
If it is in backup state and PPPOE is not active, then it has internet connectivity via pfsense-b.
After migrating to pfSense Plus 23.01, this isn't working anymore.
During bootup PPPOE is not active, so the default gateway is set to VLAN002_GATEWAY.
# netstat -rn4 -F0 | grep default default link#7 US ixl0.2
After PPPOE is up, the behavior is kind of strange:
- As long as VLAN002_GATEWAY exists in the configuration, changing the default gateway to WAN_PPPOE or WAN_GROUP always produces the same result:
The default gateway will always be set to ixl0.2, i.e. VLAN002_GATEWAY, no matter which Tier level I set it to and independent of the UI configuration options regarding gateway monitoring/actions.
- Sometimes it tried to set it to WAN_PPPOE, but failed to do so.
Spelunking in the code pointed me to route_add_or_change and I added log_error to "/sbin/route add/del" calls. It then showed me:
But that didn't have any effect. After some hours of investigating I found the problem:
Mar 23 21:22:00 php-fpm 88680 /rc.filter_configure_sync: route_add_or_change(default, <MY-PPPOE-ADDRESS-REMOVED>, , , inet) Mar 23 21:22:00 php-fpm 88680 /rc.filter_configure_sync: /sbin/route del -inet default -iface ixl0.2 Mar 23 21:22:00 php-fpm 88680 /rc.filter_configure_sync: route_del(default, inet) Mar 23 21:22:00 php-fpm 88680 /rc.filter_configure_sync: /sbin/route add -inet default <MY-PPPOE-ADDRESS-REMOVED>
Removing the default route with the following command worked:
[root@pfsense-a ~]# route del -inet default -iface ixl0.2 route: route has not been found del net default: gateway ixl0.2 fib 0: not in table [root@pfsense-a ~]# route show -inet default route to: default destination: default mask: default fib: 0 interface: ixl0.2 flags: <UP,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 [root@pfsense-a ~]# route show -inet default -iface ixl0.2 route to: default destination: default mask: default fib: 0 interface: ixl0.2 flags: <UP,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0
Now the route was gone. However, after making a change to the description of a gateway, something that has no effect on the configuration, and pressing Apply in the UI again, it, for some strange reason, set the default gateway back to ixl0.2.
[root@pfsense-a ~]# route del -inet default
- Only if I remove VLAN002_GATEWAY from the configuration completely and manually remove the ixl0.2 default gateway with the above command, will it successfully be able to set the default gateway to WAN_PPPOE. Then the globe symbol also shows up in the UI next to WAN_PPPOE.
Could someone please take a look?
Updated by Jim Pingle 2 months ago
- Status changed from New to Not a Bug
This is not and will not be a supported CARP configuration. WANs must be static using CARP for CARP to function properly (no PPPoE WANs, no DHCP WANs, etc). Routing through the other node is not supported in this manner.