pfSense » pfSense Plus

Hi,

the following setup is working just fine on pfSense CE 2.6.0:

• High Availability/CARP
• Gateway group WAN_GROUP consisting of:

• WAN_PPPOE Tier 1
• VLAN002_GATEWAY Tier 5
• Trigger level: Member down

• VLAN002_GATEWAY points to CARP VIP in VLAN 2
• VLAN 2:

• pfsense 10.0.2.1 (CARP VIP)
• pfSense-a 10.0.2.2
• pfSense-b 10.0.2.3

• Default IPv4 Gateway: WAN_GROUP
• Default IPv6 Gateway: WAN_DHCP6

This should have the following effect:

If pfsense-a is CARP MASTER and has PPPOE active, then the default ipv4 gateway is WAN_PPPOE.
If it is in backup state and PPPOE is not active, then it has internet connectivity via pfsense-b.

After migrating to pfSense Plus 23.01, this isn't working anymore.

During bootup PPPOE is not active, so the default gateway is set to VLAN002_GATEWAY.

# netstat -rn4 -F0 | grep default
default            link#7             US       ixl0.2

After PPPOE is up, the behavior is kind of strange:

• As long as VLAN002_GATEWAY exists in the configuration, changing the default gateway to WAN_PPPOE or WAN_GROUP always produces the same result:

  The default gateway will always be set to ixl0.2, i.e. VLAN002_GATEWAY, no matter which Tier level I set it to and independent of the UI configuration options regarding gateway monitoring/actions.

• Sometimes it tried to set it to WAN_PPPOE, but failed to do so.

  Spelunking in the code pointed me to route_add_or_change and I added log_error to "/sbin/route add/del" calls. It then showed me:

Mar 23 21:22:00     php-fpm     88680     /rc.filter_configure_sync: route_add_or_change(default, <MY-PPPOE-ADDRESS-REMOVED>, , , inet)
Mar 23 21:22:00     php-fpm     88680     /rc.filter_configure_sync: /sbin/route del -inet default -iface ixl0.2
Mar 23 21:22:00     php-fpm     88680     /rc.filter_configure_sync: route_del(default, inet)
Mar 23 21:22:00     php-fpm     88680     /rc.filter_configure_sync: /sbin/route add -inet default <MY-PPPOE-ADDRESS-REMOVED>

But that didn't have any effect. After some hours of investigating I found the problem:

[root@pfsense-a ~]# route del -inet default -iface ixl0.2
route: route has not been found
del net default: gateway ixl0.2 fib 0: not in table

[root@pfsense-a ~]# route show -inet default
   route to: default
destination: default
       mask: default
        fib: 0
  interface: ixl0.2
      flags: <UP,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

[root@pfsense-a ~]# route show -inet default -iface ixl0.2
   route to: default
destination: default
       mask: default
        fib: 0
  interface: ixl0.2
      flags: <UP,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Removing the default route with the following command worked:

[root@pfsense-a ~]# route del -inet default

Now the route was gone. However, after making a change to the description of a gateway, something that has no effect on the configuration, and pressing Apply in the UI again, it, for some strange reason, set the default gateway back to ixl0.2.

• Only if I remove VLAN002_GATEWAY from the configuration completely and manually remove the ixl0.2 default gateway with the above command, will it successfully be able to set the default gateway to WAN_PPPOE. Then the globe symbol also shows up in the UI next to WAN_PPPOE.

Could someone please take a look?