Regression #14171


High Availability Setup with Gateway to secondary pfSense not working - No Internet

Added by name name 2 months ago. Updated 2 months ago.

Not a Bug
Target version:
Start date:
Due date:
% Done:


Estimated time:
Release Notes:
Affected Plus Version:
Affected Architecture:



the following setup is working just fine on pfSense CE 2.6.0:

  • High Availability/CARP
  • Gateway group WAN_GROUP consisting of:
  • WAN_PPPOE Tier 1
  • VLAN002_GATEWAY Tier 5
  • Trigger level: Member down
  • VLAN002_GATEWAY points to CARP VIP in VLAN 2
  • VLAN 2:
  • pfsense (CARP VIP)
  • pfSense-a
  • pfSense-b
  • Default IPv4 Gateway: WAN_GROUP
  • Default IPv6 Gateway: WAN_DHCP6

This should have the following effect:

If pfsense-a is CARP MASTER and has PPPOE active, then the default ipv4 gateway is WAN_PPPOE.
If it is in backup state and PPPOE is not active, then it has internet connectivity via pfsense-b.

After migrating to pfSense Plus 23.01, this isn't working anymore.

During bootup PPPOE is not active, so the default gateway is set to VLAN002_GATEWAY.

# netstat -rn4 -F0 | grep default
default            link#7             US       ixl0.2

After PPPOE is up, the behavior is kind of strange:

  • As long as VLAN002_GATEWAY exists in the configuration, changing the default gateway to WAN_PPPOE or WAN_GROUP always produces the same result:

    The default gateway will always be set to ixl0.2, i.e. VLAN002_GATEWAY, no matter which Tier level I set it to and independent of the UI configuration options regarding gateway monitoring/actions.

  • Sometimes it tried to set it to WAN_PPPOE, but failed to do so.

    Spelunking in the code pointed me to route_add_or_change and I added log_error to "/sbin/route add/del" calls. It then showed me:

Mar 23 21:22:00     php-fpm     88680     /rc.filter_configure_sync: route_add_or_change(default, <MY-PPPOE-ADDRESS-REMOVED>, , , inet)
Mar 23 21:22:00     php-fpm     88680     /rc.filter_configure_sync: /sbin/route del -inet default -iface ixl0.2
Mar 23 21:22:00     php-fpm     88680     /rc.filter_configure_sync: route_del(default, inet)
Mar 23 21:22:00     php-fpm     88680     /rc.filter_configure_sync: /sbin/route add -inet default <MY-PPPOE-ADDRESS-REMOVED>
But that didn't have any effect. After some hours of investigating I found the problem:
[root@pfsense-a ~]# route del -inet default -iface ixl0.2
route: route has not been found
del net default: gateway ixl0.2 fib 0: not in table

[root@pfsense-a ~]# route show -inet default
   route to: default
destination: default
       mask: default
        fib: 0
  interface: ixl0.2
      flags: <UP,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

[root@pfsense-a ~]# route show -inet default -iface ixl0.2
   route to: default
destination: default
       mask: default
        fib: 0
  interface: ixl0.2
      flags: <UP,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
Removing the default route with the following command worked:
[root@pfsense-a ~]# route del -inet default
Now the route was gone. However, after making a change to the description of a gateway, something that has no effect on the configuration, and pressing Apply in the UI again, it, for some strange reason, set the default gateway back to ixl0.2.
  • Only if I remove VLAN002_GATEWAY from the configuration completely and manually remove the ixl0.2 default gateway with the above command, will it successfully be able to set the default gateway to WAN_PPPOE. Then the globe symbol also shows up in the UI next to WAN_PPPOE.

Could someone please take a look?

Actions #1

Updated by name name 2 months ago

I forgot to add: All currently available patches were applied via the System Patches package, before any testing was done.

Actions #2

Updated by Jim Pingle 2 months ago

  • Status changed from New to Not a Bug

This is not and will not be a supported CARP configuration. WANs must be static using CARP for CARP to function properly (no PPPoE WANs, no DHCP WANs, etc). Routing through the other node is not supported in this manner.


Also available in: Atom PDF