pfSense » pfSense Packages

Ive ran into a similar issue as well. The routes will appear in FRR but you check the pfsense route table the routes are never there.
Usually a 'clear ip bgp neighbor x.x.x.x" does the trick but this is to intrusive if this is a production peer.

please upgrade pf23.09 and frr 8.5.2 for test

updated to CE 2.7.1, FRR 2.0.2, WireGuard 0.2.1 - the issue is still persist.

CE 2.7.2, FRR 2.0.2_1 (frr9-9.0.2), WireGuard 0.2.1 - still the same.

Still reproducible on 2.8.1-RELEASE. I have root-caused this, and it is not actually
WireGuard-specific -- I have filed the underlying issue as #16963 (pfSense core > Interfaces).

What is actually happening¶

The WireGuard interface is never touched. Reconfiguring an assigned interface re-applies its IP
address even when nothing about it has changed, briefly removing and re-adding it (~3 ms apart).
While the address is gone, the kernel purges every route whose nexthop resolved through that subnet
-- and emits no routing-socket message for those purged routes. zebra is therefore never told, keeps
them flagged "B>*", and never reinstalls them.

The BGP session does not notice either: the interface never goes down (zero RTM_IFANNOUNCE), so
fast-external-failover does not fire, and a 3 ms blink is far below the hold timer. Nothing
withdraws, so nothing re-advertises.

The specific line is pfSense_interface_setaddress() at interfaces.inc:4392, confirmed by
instrumenting the file and tracing at runtime. A bare interface_configure("optX") -- i.e.
Interfaces > OPTx > Save > Apply, with the WireGuard package not involved at all -- reproduces it
identically, which is why I filed it against core rather than only here.

Why it has been hard to pin down¶

Restarting the WireGuard service does NOT reproduce it. That path destroys the interfaces, which
resets every BGP session, which triggers BGP's own withdraw/re-advertise and papers over the fault.
The gentle GUI Apply is the damaging one precisely because it is quiet enough that nothing notices.

It also never self-heals. I left a stranded route for 8 minutes (8 keepalive intervals) with
"show bgp summary" reporting Established and the correct PfxRcd the whole time, while the remote
network was unreachable. Only "netstat -rn" reveals it.

Workaround¶

A hard "clear bgp <peer>" restores the routes, as noted earlier in this thread. It works because it
forces the withdrawal that never happened: bgpd withdraws every route from that peer -- a
protocol-level delete zebra does honor -- and reinstalls them when the session re-establishes.

Worth knowing: the soft clear does not work. "clear bgp <peer> soft in" is a no-op here, because
bgpd's best path is unchanged, so it sends zebra nothing at all.

Automating the workaround¶

Rather than restarting FRR on a timer, this can be driven off the event itself. devd does see the
address change even though there is no interface event, so a rule on IFNET/ADDR_ADD is enough.

/usr/local/etc/devd/frr-fib-heal.conf

notify 100 {
        match "system"          "IFNET";
        match "type"            "ADDR_ADD";
        action "/usr/local/bin/frr-fib-heal.sh $subsystem";
};

/usr/local/bin/frr-fib-heal.sh (mode 0755)

#!/bin/sh
# Triggered by devd on IFNET ADDR_ADD. Clears only the BGP neighbors that live on the
# interface whose address just changed. No-op if that interface has no BGP peer, or if
# FRR is not installed.
IF="$1" 
[ -n "$IF" ] || exit 0

VTYSH=/usr/local/bin/vtysh
[ -x "$VTYSH" ] || exit 0

for peer in $("$VTYSH" -c 'show bgp ipv4 unicast summary' 2>/dev/null | awk '/^[0-9]+\./{print $1}'); do
        nhif=$(route -n get "$peer" 2>/dev/null | awk '/interface:/{print $2}')
        if [ "$nhif" = "$IF" ]; then
                "$VTYSH" -c "clear bgp $peer" >/dev/null 2>&1
                logger -t frr-fib-heal "ADDR_ADD on ${IF}: cleared bgp neighbor ${peer}" 
        fi
done
exit 0

Note that devd reads its configuration only at startup and has no reload mechanism -- SIGHUP
terminates it -- so a reboot is needed after adding the rule. ("service devd restart" is not safe
on pfSense: /etc/rc.d/devd sets command="/sbin/devd" with no flags, so it would relaunch with the
default /etc/devd.conf instead of /etc/pfSense-devd.conf.)

This heals in about 3 seconds. Sampling kernel routes every 0.2 s across an Apply shows the whole
cycle: 9 routes -> 1 route -> 9 routes, with the hook firing in between.

Full analysis in #16963.

Great job, Dan!
Hopefully it will get fixed in CE