pfSense

Seeing what I think is a related (though different) issue in 2.2.4

DDNS set to monitor gateway group {WAN1_WAN2_FAILOVER} (use WAN1, failover to WAN2)

On forcing WAN1 Gateway down via System->Routing, hosts failover to WAN2 connection, however DDNS shows WAN1 IP in red (stale entry)

Leaving in this state for ~30 minutes, DDNS never updates to reflect WAN2 address

Performing "Save & Force Update", DDNS updates to reflect WAN2 address as expected

On restoring WAN1 Gateway via System->Routing, new connections are established over WAN1, however DDNS still reflects WAN2 IP

Leaving in this state for ~60 minutes, DDNS never updates until once again forced

This seems to be an issue with Gateway state changes not properly triggering DDNS updates

WAN0 shows as down and WAN1 up.
I'm beginning to suspect that something in the kernel went wonky because I believe queues also started going sideways as well. Will try to reproduce the issue this weekend and post results.
FYI, when I did test this the weekend after the outage, it reliably did not update DDNS when WAN0 went down, but reliably did update when WAN0 came back up. DDNS just never updated to WAN1 IP.

Okay, I'm not sure why this changed, as I've made no changes to config since opening the ticket, but I noticed this while testing tonight.
Looks to me like the script used to determine the active interfaces public IP get's confused if we're NAT'd. GWCP has an IP of 192.168.16.2, which goes through a router at 192.168.16.1, which provides WAN access via a 4G connection. I'm positive this wasn't an issue a few releases back, as our primary connection goes down all the time and all our VOIP phones would fail if the DDNS update didn't work.

Here's what I'm seeing in the log now:

Oct 9 21:25:48    php-fpm[8906]: /services_dyndns.php: Default gateway down setting GWCP as default!
Oct 9 21:25:47    php-fpm[8906]: /services_dyndns_edit.php: DynDNS (cranesmartvoip.ddns.net) There was an error trying to determine the public IP for interface - balance0(em2_vlan11). Probably interface is not a WAN interface.
Oct 9 21:25:47    php-fpm[8906]: /services_dyndns_edit.php: MONITOR: GWtg is down, omitting from routing group balance0
Oct 9 21:25:47    php-fpm[8906]: /services_dyndns_edit.php: Default gateway down setting GWCP as default!
Oct 9 21:25:47    php-fpm[8906]: /services_dyndns_edit.php: DynDNS (cranesmartvoip.ddns.net): running get_failover_interface for balance0. found em2_vlan11
Oct 9 21:25:47    php-fpm[8906]: /services_dyndns_edit.php: MONITOR: GWtg is down, omitting from routing group balance0
Oct 9 21:25:47    php-fpm[8906]: /services_dyndns_edit.php: Default gateway down setting GWCP as default!
Oct 9 21:25:47    php-fpm[8906]: /services_dyndns_edit.php: Dyndns debug information (cranesmartvoip.ddns.net): Could not resolve checkip.dyndns.org to IP using interface IP 192.168.16.2.
Oct 9 21:25:34    php-fpm[8906]: /services_dyndns_edit.php: MONITOR: GWtg is down, omitting from routing group balance0
Oct 9 21:25:34    php-fpm[8906]: /services_dyndns_edit.php: Default gateway down setting GWCP as default!
Oct 9 21:25:34    php-fpm[8906]: /services_dyndns_edit.php: DynDns: updatedns() starting

In that last log snippet, you didn't have functional DNS resolution. That "Could not resolve checkip.dyndns.org" log means gethostbyname got no valid response after 3 attempts.

Yup, I saw that and dug through the code and believe I've figured out what's going on (although I seem to have two separate issues... the lookup issue is the one I've figured out).

I had 4 DNS servers configured for our system. The first two are from our primary ISP, and they only allow lookups from their customers. The other two are 8.8.8.8 and 8.8.4.4. 8.8.8.8 was also my monitor IP for WAN0 while 8.8.4.4 was the monitor IP for WAN1. Because 8.8.4.4 is the monitor IP for WAN1, and pfSense uses DNS servers in a round robin fashion, 8.8.4.4 was being used for lookups. Because it's also the monitor IP for WAN1, those lookups would go over WAN1, even when WAN0 was still up (firewall rules force all traffic destined for 8.8.4.4 out via WAN1) (gateway monitoring).
Because WAN1 is a 4G, and is exorbitantly expensive, I decided to disable DNS lookups for 8.8.4.4, which brought WAN1 traffic down to 0 when it was not in use.

Incidentally however, I'd forgotten that my other three remaining DNS servers were either bound to WAN0's ISP, or 8.8.8.8 (monitor IP). Thus, when WAN0 went down, we had no DNS servers available. I guess I didn't notice this because we also have an internal DNS, so no one ever had any issue with name resolution, even though pfSense did.

I'm still using 8.8.4.4 for WAN1 monitor, but added a fourth DNS server, and failover and back now seems to work. It still doesn't answer the original problem, but that problem seems to have magically resolved itself.
I'll continue monitoring and come back here if the original issue pops up again, but for now, all seems smooth.

Thanks for the feedback. The original issue doesn't always happen, so as long as the config is otherwise fine, you might only hit that 1 in 10 or 20 or more tries.