pfSense

If I set the Tier 1 gateway as "Mark Gateway as Down" then turn that setting back off, the routing will correct itself and switch back to the Tier 1 gateway.

To make things even more complicated, in the workaround mentioned above, the routing actually changes back to the Tier 1 gateway when I mark it as down, so that when the status is "forced offline" it is still routing through that gateway. When I undo the "mark as down" it continues to route through that gateway.

The Gateway Group was set as Trigger Level: Packet Loss or High Latency. I changed that to "Member Down" and now the routing seems to be switching back to the Tier 1 gateway as it should. I'm going to revert to "Packet Loss or High Latency" as a test to see if that triggers the problem.

With the Gateway Group set to "Packet Loss or High Latency" this problem definitely shows up much more often.

The same problem on all my devices (20 devices) after upgrading

@VasylSemenchuk Are your gateway groups set to trigger level "Packet Loss or High Latency" or "Member Down"? Does it work better if set to "Member Down"?

Set to trigger level "Packet Loss or High Latency"
I will set trigger level "Member Down" and let you know on monday or thursday

Did you try restart service dpinger? In my case this helps switch back to WAN1

Also i noticed in my case helps when restart openvpn client.
After restart OpenVPN, vpn and other traffic switch back to WAN1

HI! After some tests noticed that problem appear only when my "Gateway Group" set as Default gateway
If set WAN1 or WAN2 as Default gateway, switching working fine

Generally same problem here, BUT EVEN HAVE PROBLEM WHEN THE GATEWAY GROUP ISN'T PFSENSE DEFAULT GATEWAY. After digging a little bit deeper, I found that the problem resides in failing to generate correct config in /tmp/rules.debug. As f ar as I see, there is no clue shows that there is problem with pfctl reading rules; therefore, I thought the problem might be in parts that take charg e of generating rules or calls generating. For example, when alarms on loss or delay disappeared, pfsense didn't call rules generating process.
I currently try only member down as trigger, and I'd like to know if it works for you.
As I noticed, when problems present, a simple filter reload will get everything back to normal. I don't know if it will work for you. If it works, a temporary workaround may be some cron runned script monitoring if gateway is correct.

Mitch Claborn wrote:

@VasylSemenchuk Are your gateway groups set to trigger level "Packet Loss or High Latency" or "Member Down"? Does it work better if set to "Member Down"?

Do you have problem if the trigger is member down only?

Vasyl Semenchuk wrote:

Did you try restart service dpinger? In my case this helps switch back to WAN1

Actually dpinger is pretty reliable in giving gateway status.

Any update on this? I experience the very same problem on version 2.4.4-RELEASE-p3 (amd64).

Seems related to #10716

I am not sure if my issue:

https://forum.netgate.com/topic/156890/dpinger-broken-or-dashboard-broken-or-my-brain-is-broken?_=1600704062714

is related to that what you describe.

If not, I would open a seperate bug ticket, if not, I would contribute information and further testing if needed.

Right now, for me, Multi-WAN is working, but not able to recover from 100% packet loss events.

I have the same issue here.
Once pfSense switches to the Tier 2 gateway, the only way to make it come back to the Tier 1 is by disabling ("Never") and reenabling the Tier 2 gateway.

Viktor Gurov wrote:

no such issue on 2.5.0.a.20201022.1850, resolved in #10716
failover and load-balance gw groups tested

When version 2.5.0 is going to be released? Any roadmap on that?

Now tested with 2.5.0.a.20201101.1850

I still get for unknown reasons sometimes partial or full loss for alive-ping at one of the 2 WAN interfaces, but this is not the issue.

Nov 2 10:37:56 dpinger 16236 WAN_PHY1_IGB0GW 8.8.4.4: Alarm latency 0us stddev 0us loss 100%

Problem is that this status remains until any change to the gateway group is made - then it works immediately.

dpinger is not reattempting to reach the defined IP or the process maintaining the operational status is not taking over the changes.

https://forum.netgate.com/topic/156890/dpinger-broken-or-dashboard-broken-or-my-brain-is-broken/23

Jörn Greszki wrote:

Now tested with 2.5.0.a.20201101.1850

I still get for unknown reasons sometimes partial or full loss for alive-ping at one of the 2 WAN interfaces, but this is not the issue.

Nov 2 10:37:56 dpinger 16236 WAN_PHY1_IGB0GW 8.8.4.4: Alarm latency 0us stddev 0us loss 100%

Problem is that this status remains until any change to the gateway group is made - then it works immediately.

dpinger is not reattempting to reach the defined IP or the process maintaining the operational status is not taking over the changes.

https://forum.netgate.com/topic/156890/dpinger-broken-or-dashboard-broken-or-my-brain-is-broken/23

Please create new redmine issue for that
may be related to #8136