pfSense

I neglected to mention that I was using "Disable Gateway Monitoring Action" on my gateways when the above issues occurred. Perhaps that isn't relevant. In any event, I've re-enabled actions because I'm now using a Gateway Group.

David Myers I believe I'm facing this exact issue, take a look at https://forum.netgate.com/topic/172849/rtsold-not-running-ipv6-wan-dhcp-keeps-losing-connectivity/

I'm highly motivated to try to get this fixed!

It appears we are out of luck on having devd fire events for IP address changes. There is a commit: https://reviews.freebsd.org/rGa75819461ec7c7d8468498362f9104637ff7c9e9 but that seems like it might not even be in FreeBSD13 (much less 12.3...). Maybe we can get this backported to 12.x somehow? Seems like it would be immensely useful in these cases.

My poor man's solution is hashing the output of ifconfig on all DHCP6 interfaces with cron and firing an event if the hash changes. But this places unnecessary load on the system and also has a potential for up to 1 minute of downtime before the change is picked up. Still, it's the best I can come up with for now until devd can trigger on address changes. I have a PR for this that I will submit later today.

Just updated PR #4595 with the new mitigation changes. Testers & feedback wanted.

I posted on the PR that since Reid Linnemann has just deprecated pfSense_getall_interface_addresses(), this should probably be updated to use the new pfSense_get_ifaddrs() function, where possible.

However, I do want to note for the record that my patch has been running for over a week now and has 100% fixed my issue.

Jim Pingle wrote in #note-15:

Still waiting on feedback from someone who can reproduce this to test against a 2.7.0 snap, 23.01 release, or a 23.05 snap.

I'm pretty sure this is still happening on 23.01. With T-Mobile Home Internet I can go weeks without the IPv6 address changing, or it can change several times in one day, so I can't reproduce it on demand.

I've written a simple-minded script to detect and correct the problem and run it from cron every 5 minutes, and there's evidence in the logs of it taking action a few times.

#!/usr/bin/env perl
#
# Delete deprecated IPv6 addresses.
#
use v5.30;

my $if = 'igb2';
my $deleted = 0;

open(my $ifconfig, "-|", "/sbin/ifconfig $if inet6") || die "Can't run /sbin/ifconfig: $!";
while (<$ifconfig>) {
    if (/inet6 ([0-9a-fA-F.:]+) prefixlen (\d+) .*deprecated/) {
        system("/sbin/ifconfig $if inet6 $1/$2 delete");
        $deleted++;
    }
}
close($ifconfig);

if ($deleted) {
    system("/etc/rc.newwanipv6 $if");
}

In fact I think this was the script firing off early this morning. I've edited out messages from NUT:

Apr 17 01:21:49 router rc.gateway_alarm[72711]: >>> Gateway alarm: TMHI_DHCP6 (Addr:2620:fe::10 Alarm:1 RTT:53.447ms RTTsd:9.178ms Loss:41%)
Apr 17 01:21:49 router check_reload_status[401]: updating dyndns TMHI_DHCP6
Apr 17 01:21:49 router check_reload_status[401]: Restarting IPsec tunnels
Apr 17 01:21:49 router check_reload_status[401]: Restarting OpenVPN tunnels/interfaces
Apr 17 01:21:49 router check_reload_status[401]: Reloading filter
Apr 17 01:21:51 router php-fpm[65938]: /rc.filter_configure_sync: MONITOR: TMHI_DHCP6 has packet loss, omitting from routing group TMHI_FAILOVER6
Apr 17 01:21:51 router php-fpm[65938]: 2620:fe::10|2607:fb90:7522:c4a9:a484:c093:675:c336|TMHI_DHCP6|52.851ms|8.805ms|44%|down|highloss
Apr 17 01:25:00 router php-cgi[83652]: rc.newwanipv6: rc.newwanipv6: Info: starting on igb2.
Apr 17 01:25:00 router php-cgi[83652]: rc.newwanipv6: rc.newwanipv6: on (IP address: 2607:fb90:759b:d695:2e0:67ff:fe26:47ea) (interface: opt1) (real interface: igb2).
Apr 17 01:25:01 router php-cgi[83652]: rc.newwanipv6: Removing static route for monitor 2620:fe::fe:10 and adding a new route through fdfd:10:167:100::1
Apr 17 01:25:01 router php-cgi[83652]: rc.newwanipv6: Removing static route for monitor 9.9.9.10 and adding a new route through 192.168.12.1
Apr 17 01:25:01 router php-cgi[83652]: rc.newwanipv6: Removing static route for monitor 2620:fe::10 and adding a new route through fe80::c6e5:32ff:fed7:634e%igb2
Apr 17 01:25:01 router php-cgi[83652]: rc.newwanipv6: Removing static route for monitor 149.112.112.10 and adding a new route through 10.155.98.1
Apr 17 01:25:02 router check_reload_status[401]: Reloading filter
Apr 17 01:25:02 router php-cgi[83652]: rc.newwanipv6: The command '/sbin/ifconfig igb2 inet6 2607:fb90:7522:c4a9:2e0:67ff:fe26:47ea delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address' 
Apr 17 01:25:02 router php-cgi[83652]: rc.newwanipv6: Resyncing OpenVPN instances for interface TMHI.
Apr 17 01:25:02 router php-cgi[83652]: rc.newwanipv6: Creating rrd update script
Apr 17 01:25:02 router php-cgi[83652]: rc.newwanipv6: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 2607:fb90:7522:c4a9:2e0:67ff:fe26:47ea -> 2607:fb90:759b:d695:2e0:67ff:fe26:47ea - Restarting packages.
Apr 17 01:25:02 router check_reload_status[401]: Starting packages
Apr 17 01:25:02 router check_reload_status[401]: Reloading filter
Apr 17 01:25:03 router php-fpm[79152]: /rc.start_packages: Restarting/Starting all packages.
Apr 17 01:25:03 router php-fpm[79152]: /rc.start_packages: Stopping service nut
Apr 17 01:25:03 router php-fpm[79152]: /rc.start_packages: Starting service nut
Apr 17 01:25:03 router php-fpm[7842]: /rc.filter_configure_sync: MONITOR: TMHI_DHCP6 is available now, adding to routing group TMHI_FAILOVER6
Apr 17 01:25:03 router php-fpm[7842]: 2620:fe::10|2607:fb90:759b:d695:2e0:67ff:fe26:47ea|TMHI_DHCP6|43.574ms|1.428ms|0.0%|online|none

The bug is still present.
I am able to reproduce this on our Netgate 6100 running 24.11 with an "AVM Fritz!Box 6850 LTE" as 4G router in front of it.

• RA is sent by the AVM router
• A new GUA is set on the interface and the old one is marked as deprecated:
  

  # ifconfig igc3
  igc3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
          description: LAN4_WAN_LTE
          options=4e020bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
          ether 90:ec:77:29:88:eb
          inet 192.168.178.10 netmask 0xffffff00 broadcast 192.168.178.255
          inet6 ***AVM LLA***%igc3 prefixlen 64 scopeid 0x4
          inet6 ***OLD GUA*** prefixlen 64 deprecated autoconf pltime 0 vltime 7191
          inet6 ***NEW GUA*** prefixlen 64 autoconf pltime 3600 vltime 7200
          media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
  

• No link up/down event happens
• IPv4 connection heals itself

The only logs I could find recognizing the IPv6 change had been:

/var/log/ipsec.log:Jan 11 00:11:12 hamfw1 charon[34587]: 09[KNL] ***NEW GUA*** appeared on igc3
/var/log/ntpd.log:Jan 11 00:11:14 hamfw1 ntpd[93745]: Listen normally on 108 igc3 [***NEW GUA***]:123

Deleting the deprecated ips and manually triggering rc.newwanipv6 still works, but the bug is still present.

For the moment I am using a variation of David Myers idea in #note-16 in a cron job.
But as → luckman212 already pointed out in #note-7 [...] this places unnecessary load on the system and also has a potential for up to 1 minute of downtime [...]

I dug a little deeper on how the renewal of delegated prefixes should happen. DHCPv6 does not have a mechanism where the DHCPv6 server can actively tell the client to renew it's leases. pfSense has to infer this from an upstream router's RA if it announces a new prefix. Then pfSense has to proactively renew it's delegated prefixes.
This implies, that the dhcp6c script is not initially involved, when a prefix rotation by an upstream router happens. pfSense has to for instance actively pkill -HUP dhcp6c to force dhcp6c initialte a new request for the script to be run afterwards. (And make all necessary reconfiguration happen)

What we need, I think, is a hook directly in the kernel which is triggered if a new prefix announcement arrives via RA. This then sould restart dhcp6c(?), so dhcp6c immediately sends out a new request. AFAIK does dhcp6c not react to RA on it's own. I've allready written some dyndns-deamons on Linux who subscribe to kernel netlink events to grasp certain route changes but I don't know yet if BSD comes with a similar facility to get notified when the kernel received new route information via RA.

Currently there seem to be no procedure in place which forces dhcp6c to request new prefixes if an RA with new prefixes or invalidated (lifetime=0) prefixes arrives.

I'am quiet confused.

On a WAN interface with "Do not wait for RA" is NOT checked (so a WAN interface where we expect RA), an rtsold instance should be spawned. rtsold should execute /var/etc/rtsold_vtnet1_script.sh whenever an RA is received. rtsold_vtnet1_script.sh should kill dhcp6c whenever an RA resulted in new ip addresses on that WAN interface.

As far as I can see, pfsense does it the other way around. It starts rtsold only when "Do not wait for RA" is checked. The part which kills dhcp6c on ip change in /var/etc/rtsold_vtnet1_script.sh is only present when "Do not wait for RA" is NOT checked. That is, when rtsold is not started.

Sadly rtsold daemon always dies quickly after it is being started . So pfsense can not react to new prefixes anyways. I have not yet found what causes rtsold to die.

I wonder if there is some issue in the logic when to start rtsold.

When "Do not wait for RA" is checked:
rtsold schould send solicitation requests to force upstream routers respond with a unicast RA.
Regardless of "Do not wait for RA":
rtsold must always listen for RA events to be able to react to upstream prefix changes. <- that is not happening and likely the cause of trouble

rtsold_vtnet1_script.sh should kill dhcp6c whenever an RA resulted in new ip addresses on that WAN interface.

My understanding is RA and dhcp6c can work at the same time. A deprecated prefix from RA does not necessarily mean the DHCP lease is invalid. DHCPv4 has the same issue with subnet changes - there's no mechanism in DHCP itself to inform clients of a change. With IPv6, a prefix change in DHCP should be supplemented with an RA message to inform clients of new and deprecated prefixes.

And yes, the plan is to have the updated function in CE 2.8.

I am also looking into this issue.
There are different possible issues here, depnding on configuration method.

1. startup or configure interface
2. rtsold send a router solicitation message to get a quick route advertisement (ok)
3. FreeBSD kernel processes the RA [assigns address to interface, sets route] (ok)
4. dhcp6c is started for other configuration (ok)
5. dhcp6c renew interval (specified by DHCP server in T1, 30 minutes for my FritzBox) runs (ok)
6. provider assigns new IPv6 subnets to WAN router,i.e. FritzBOX (ok)
7. WAN router sends unsolicited route advertisment with new prefix (ok)
8. FreeBSD kernel processes the RA [assigns address to interface, sets route] (ok)
   note: neither dhcp6c is made aware of this nor pfsense is made aware of this change so no immediate action besides kernel updateing the ifconfig and route [FreeBSD Kernel does not handle the Managed and OtherConfiguration flags of an RA] (here must be the primary fix for this case - trigger dhcp6c Request - or successfull renew or rebind, see next point)
9. on next renew interval (T1), dhcp6c issues a RENEW and gets back an Info with a message level "status code: no binding" - but does not properly handle it [is not directly defined in the RFC]. It only prints it to log, but neither does the REQUEST [implemented in, for example, an opensense fork of the dhpc6c, but not the freebsd branch of hrs-allbsd/wide-dhcpv6] nor at least hands this information the script, which could initiate a REQUEST instead. The internal Lease state stays ACTIVE. (again no action taken here to fix the address issue)
10. because renew failed at rebind interval time (T2), dhcp6c issues a REBIND for the IA - with basically the same results as the RENEW - just the refresh timer is cancelled. (again no action taken here to fix the address issue)
11. validlifteime of address expires (2 hours after last successfull renew in my case), dhcp6c releases it and starts over (here everything recovers and begins at 5...)

It should be possible to add the patch regarding global "status code: no binding" to the dhcpc6 port in addition to the existing pfsense added patch. And there is the possibility to react on added interface addresses via devd integration in the meantime, so at least for simple setups as mine it should be quite simple to implement.