pfSense

It shows:

Name: WAN
Gateway: Changes from GW IP to "Dynamic"
Monitor: GW IP
RTT: 0.5ms
Loss: Changes from 0% to 100%
Status: Changes from Online to Offline

The one for ppp0 remains Online with 0% Loss

My setup for the WAN looks like this:

ISP - switch - Pfsense

If I pull the cable between the ISP and the switch instead, the gateway status changes just like above (Gateway does not change from GW IP, to "Dynamic" but remains the GW IP). When I tcpdump the re0 (WAN) interface I can see that pfsense still sends out the traffic to the WAN despite it being Offline.

check the output of:

grep route-to /tmp/rules.debug

is it updating the rules to the correct gateway?

The gateway group with re0 and ppp0 still says " route-to { ( re0 x.x.x.x ) } "

what if you run:

/etc/rc.filter_configure_sync

then check again, which gateway is route-to showing?

First of all when I run /etc/rc.filter_configure_sync it outputs 16 of these:
Warning: dns_get_record(): DNS Query failed in /etc/inc/notices.inc on line 390

And when it is actually done the gateway group says:
" route-to { ( ppp0 y.y.y.y ) } "

And the traffic is now output on the right interface.

This might be a DNS related issue dues to the warning messages I get. So FYI I am using the DNS Resolver with these settings.

No DNS servers in System->General Setup.
Forwarding Mode Disabled
Network Interfaces are set to LAN
Outgoing Network interfaces are set to re0 (WAN) and ppp0.

Maybe I misunderstood something here that makes things break?

Warning: dns_get_record(): DNS Query failed in /etc/inc/notices.inc on line 390
That comes from trying to look up the name of where you want to send Growl notifications. IMHO it should not be relevant to the failover issue here. You could disable Growl notifications to shut that up.

On the issue at hand, for some reason filter_configure_sync has not run (or not succeeded) after the fail event. I will let Chris with the fault-finding, which he might prefer to be on the forum until the real root cause is determined.

First of all i turned off Grown notifications (thought they were off). And the warnings disappeared. Thanks!

I tried to disconnection of the WAN interface again. And it worked.. Once..

However I'm seeing the same issue as #4121. So there is something fishy with the failover functionality. But every time now when I have issues if I run

 /etc/rc.filter_configure_sync 

Maybe the script is run too early or isn't run often enough to make it work in all scenarios. I will do more debugging this evening.

If you want me to me this to the forum just let me know.

I've tried back and forth to trigger the fault again, but it is a no show, which is both good and bad. :)

However I still have the issue with the OpenVPN connection which does not trigger update of the gateway group containing:

ovpnc7 Tier 1 (Connects through re0)
re0 Tier 2 (This is WAN)
ppp0 Tier 3 (Backup internet)

If I run the rc.filter_configure_sync while the VPN is up, the route-to goes to ovpnc7.

If I disable the VPN configuration the route-to goes to re0. When I re-enable it it never changes to ovpnc7.

If I disconnect the WAN, the route-to goes to ppp0. If I re-enable the WAN, and wait for the VPN to go up, it still sets it to re0 and not ovpnc7.

I do not know if there is another bug open for that. However I guess this can be closed, sorry for the hassle.