pfSense

Just for completeness, I tested with -S (rather than-s) with similar result.

[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 expires in 1086 seconds [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 expires in 1086 seconds [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 1086 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 expires in 1086 seconds [ethernet]
[23.09-RELEASE][root@fw]/root: arp -S 192.168.230.201 70:b3:d5:e3:d0:22
192.168.230.201 (192.168.230.201) deleted
[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 expires in 1066 seconds [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 expires in 1066 seconds [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 1066 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 permanent [ethernet]
[23.09-RELEASE][root@fw]/root: arp -S 192.168.230.202 70:b3:d5:e3:d0:40
192.168.230.202 (192.168.230.202) deleted
[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 permanent [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 expires in 1045 seconds [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 1045 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 expired [ethernet]
[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 expires in -1699744409 seconds [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 expires in 1028 seconds [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 1028 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 permanent [ethernet]
[23.09-RELEASE][root@fw]/root: arp -S 192.168.230.203 70:b3:d5:e3:d0:23
192.168.230.203 (192.168.230.203) deleted
[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 permanent [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 permanent [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 999 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 expired [ethernet]
[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 permanent [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 permanent [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 994 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 expired [ethernet]
[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 expires in -1699744450 seconds [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 expires in -1699744450 seconds [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 987 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 permanent [ethernet]
[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 permanent [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 permanent [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 974 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 expired [ethernet]
[23.09-RELEASE][root@fw]/root: arp -a | grep leontp
leontp2 (192.168.230.202) at 70:b3:d5:e3:d0:40 on ix0 expires in 1180 seconds [ethernet]
leontp3 (192.168.230.203) at 70:b3:d5:e3:d0:23 on ix0 expires in 1180 seconds [ethernet]
leontp0 (192.168.230.200) at a0:39:75:00:00:7f on ix0 expires in 1180 seconds [ethernet]
leontp1 (192.168.230.201) at 70:b3:d5:e3:d0:22 on ix0 expires in 1180 seconds [ethernet]
[23.09-RELEASE][root@fw]/root: 

How does the kernel in 24.03 compare with that of 23.09?

Denny Page wrote:

Arp flips back and forth between reporting static arp entries as permanent or having timeouts with large negative values, and eventually looses the concept that entries are static at all.

[...]

Can anyone solve this issue? It would help me on a daily base.
I use the "permanent license" to wake up devices on remote locations.

Thanks

Johan

Is this still happening on 23.09.1?

Unfortunately yes. This is still happening in 23.09.1
If I am not mistaken it started happening two or three releases ago.

The issue I opened (#15105) was a decided to be a duplicate of this one. Just pasting in the detail I added. Personally the poor (cosmetic) output would be a much lower priority than the fact the static ARP entires are lost once the device is active on the network, meaning unicast WOL only works once i.e a functional problem. Could someone create a patch that keeps re-adding the static ARP perhaps as a workaround?

Under pfsense 2.7.2, it appears that static ARP entries can be created (for example when a host is offline) however the entries are converted into regular ARP entries (which expire) once a host is present on the network. Removing the static ARP flag from the DHCP entry, applying the configuration, and then reconfiguring it and applying again appears to recreate the static ARP entry, however, if the host is present on the network it is converted into an expiring ARP entry.

Once the (now temporary) ARP entry expires, when removing the static ARP entry via the DHCP config page, I see the following in the system log:

php-fpm /services_dhcp_edit.php: The command '/usr/sbin/arp -d '192.168.5.5'' returned exit code '1', the output was 'arp: delete 192.168.5.5: No such file or directory'

I assume this is because the static ARP entry is no longer present to be removed from the backend?

I also observe that all user added devices with static ARP (i.e not the pfSense interfaces which are consistently in permanent state) can appear in various states in ARP table diagnostics page, switching between a missing/blank status, permanent, and a huge negative expiry time progressively getting more and more negative, and the regular expiry time once the device is present on the network.

The issue presented immediately after upgrading to pfsense 2.7.2.

Thank you for confirming. The 24.03 dev snaps for plus are now available, testing on that would be appreciated (the Boot Environments feature is very handy here!).

This could be also related

https://redmine.pfsense.org/issues/15104

I am having one broadcast domain now the previous version had separated intra interface broadcast domains, I never showed intra interface layer 2 traffic until 23.09

This could, could also cause broadcast arp storms and VLAN hopping vulnerabilities. Prior versions had broken up the layer 2 broadcast domains.

Jonathan Lee wrote in #note-13:

This could be also related

https://redmine.pfsense.org/issues/15104

For my part, after reading the other report I don't see how.

Boycee . wrote in #note-11:

The issue I opened (#15105) was a decided to be a duplicate of this one. Just pasting in the detail I added. Personally the poor (cosmetic) output would be a much lower priority than the fact the static ARP entires are lost once the device is active on the network, meaning unicast WOL only works once i.e a functional problem. Could someone create a patch that keeps re-adding the static ARP perhaps as a workaround?

Under pfsense 2.7.2, it appears that static ARP entries can be created (for example when a host is offline) however the entries are converted into regular ARP entries (which expire) once a host is present on the network. Removing the static ARP flag from the DHCP entry, applying the configuration, and then reconfiguring it and applying again appears to recreate the static ARP entry, however, if the host is present on the network it is converted into an expiring ARP entry.

Once the (now temporary) ARP entry expires, when removing the static ARP entry via the DHCP config page, I see the following in the system log:

php-fpm /services_dhcp_edit.php: The command '/usr/sbin/arp -d '192.168.5.5'' returned exit code '1', the output was 'arp: delete 192.168.5.5: No such file or directory'

I assume this is because the static ARP entry is no longer present to be removed from the backend?

I also observe that all user added devices with static ARP (i.e not the pfSense interfaces which are consistently in permanent state) can appear in various states in ARP table diagnostics page, switching between a missing/blank status, permanent, and a huge negative expiry time progressively getting more and more negative, and the regular expiry time once the device is present on the network.

The issue presented immediately after upgrading to pfsense 2.7.2.

I can confirm this exact behavior you described and pfSense 2.7.2 being the origination point.

If someone can provide me with two versions as closely related in time as possible along with a reproducer I can bisect this

Christian McDonald wrote in #note-17:

If someone can provide me with two versions as closely related in time as possible along with a reproducer I can bisect this

Not exactly sure what you're asking for, but I did a pretty extensive comparison between v 2.7.0 (unaffected) and 2.7.2 (affected) and can't seem to identify a cause.

My only guess is that arp_flush() is being triggered when delete_old_routes() is triggered by another event... but nothing has been changed around that processing, from what I can find.

The only other references I've seen to similar behavior on FreeBSD are due to interface-level events, not individual leases getting renewed.

Basically someone (likely me) just needs to start producing test builds at various points in time between a known good version and a known broken version. By doing a bisect you can generally land on the exact commit where the behavior changed. The key here is having a reliable reproducer to check for said behavior.

Ideally the reproducer would be a script that can be run under automation that produces a binary result...broken or not broken. It's the best way to do root cause analysis on issues like this. We can beat around the bush trying to find the root cause organically which might save time in the long run, but a bisect is a pretty mechanical way of definitely landing at the problem commit. It's just time consuming.

Looks like this might have gotten some attention upstream, will track.

https://reviews.freebsd.org/D43983

It seems like an interim fix would be to build arp with "WITHOUT_NETLINK" defined.

Is there a workaround?

We pulled in a patch that might fix this. Check out the latest 24.03 development snapshots.

Christian McDonald wrote in #note-25:

We pulled in a patch that might fix this. Check out the latest 24.03 development snapshots.

I have the community version, is there a patch for version 2.7.2?

Michele D'Alessio wrote in #note-26:

Christian McDonald wrote in #note-25:

We pulled in a patch that might fix this. Check out the latest 24.03 development snapshots.

I have the community version, is there a patch for version 2.7.2?

No patch for 2.7.2, this patch is included in CE development snapshots though.

At the moment I cannot switch to the CE version. When will it be implemented in the free version?

This fix requires new binaries and cannot be patched on older releases, the only way to get the fix will be by upgrading to complete build which includes the updated binaries. The next scheduled release is Plus 24.03, there is no scheduled time frame for another CE release, but the fix is in development snapshot builds.

Tested with 24.03 RC -- issue appears resolved.