Bug #15516
closed
Per rule byte counter values lost across a filter reload
0%
Description
In some situations the byte counter reported by pfctl is not retained correctly across a filter reload.
It appears to be dependent on the value stored. So 4414450379 Bytes is retained correctly but 4793436239 Bytes is not:
[24.03-RELEASE][admin@4200.stevew.lan]/root: pfctl -vvsr | grep -A3 @90 @90 pass in quick on igc3 reply-to (igc3 172.21.16.1) inet proto tcp from <WAN__NETWORK:1> to 172.21.16.11 port = 5201 flags S/SA keep state (if-bound) label "USER_RULE: Allow iperf" label "id:1716563048" ridentifier 1716563048 [ Evaluations: 95 Packets: 6177197 Bytes: 4793436239 States: 2 ] [ Inserted: uid 0 pid 0 State Creations: 2 ] [ Last Active Time: Fri May 24 21:27:52 2024 ] [24.03-RELEASE][admin@4200.stevew.lan]/root: /etc/rc.filter_configure [24.03-RELEASE][admin@4200.stevew.lan]/root: pfctl -vvsr | grep -A3 @90 @90 pass in quick on igc3 reply-to (igc3 172.21.16.1) inet proto tcp from <WAN__NETWORK:1> to 172.21.16.11 port = 5201 flags S/SA keep state (if-bound) label "USER_RULE: Allow iperf" label "id:1716563048" ridentifier 1716563048 [ Evaluations: 96 Packets: 6177197 Bytes: 498468943 States: 0 ] [ Inserted: uid 0 pid 0 State Creations: 0 ] [ Last Active Time: N/A ]
The packet count remains the same.
Tested in 24.03 amd64
It does not happen if you then pass more traffic raising the byte counter above the limit and then reload the filter a second time.
Updated by Steve Y about 1 month ago
Just to add another data point, as I mentioned in https://forum.netgate.com/topic/188336/firewall-rule-counters-max-size/17 a counter of 96 GB was reset but a few at 5+ GB were not.
Updated by Jim Pingle about 1 month ago
- Plus Target Version changed from 24.07 to 24.08
Updated by Kristof Provost 20 days ago
I believe I see what's happening.
When rules are re-loaded we explicitly copy the counter state from the previous rules to the new rules. This uses the pf_counter_u64_add_protected() function (specifically introduced to work around counter_64 performance issues on 32-bit platforms, i.e. 3100), but this function takes a uint32_t as an argument.
So whenever our counter is over 32-bits we end up losing the upper bits.
This is masked to some extent because the internal counters are separated between in- and out-bound counters, but those get added up in userspace before they're displayed to users.
Updated by Mateusz Guzik 20 days ago
- Assignee set to Mateusz Guzik
I agree with the analysis.
The entire thing however showed up because of performance problems on the 3100.
It very much can be addressed with modest hassle while retaining everything, which I'm going to sort out later today or tomorrow.
Updated by Mateusz Guzik 10 days ago
- Status changed from New to Feedback
Fixed in https://gitlab.netgate.com/pfSense/FreeBSD-src/-/commit/5c71199c72c42a9836ed6f36521a7ff95e169e0b
Steve and Kristof verified the 3100 and amd64 respectively.
Updated by Steve Wheeler 10 days ago
Looks good with the patched kernel:
[24.08-DEVELOPMENT][admin@3100.stevew.lan]/root: pfctl -vvsr | grep -A3 @88 @88 pass in quick on mvneta1.4084 reply-to (mvneta1.4084 172.21.16.1) inet all flags S/SA keep state (if-bound) label "USER_RULE: Allow all ipv4+ipv6 via pfSsh.php" label "id:1652894570" ridentifier 1652894570 [ Evaluations: 360 Packets: 7086185 Bytes: 5491882806 States: 7 ] [ Inserted: uid 0 pid 0 State Creations: 11 ] [ Last Active Time: Thu Jun 20 17:48:03 2024 ] [24.08-DEVELOPMENT][admin@3100.stevew.lan]/root: /etc/rc.filter_configure [24.08-DEVELOPMENT][admin@3100.stevew.lan]/root: pfctl -vvsr | grep -A3 @88 @88 pass in quick on mvneta1.4084 reply-to (mvneta1.4084 172.21.16.1) inet all flags S/SA keep state (if-bound) label "USER_RULE: Allow all ipv4+ipv6 via pfSsh.php" label "id:1652894570" ridentifier 1652894570 [ Evaluations: 366 Packets: 7086207 Bytes: 5491885050 States: 0 ] [ Inserted: uid 0 pid 0 State Creations: 0 ] [ Last Active Time: N/A ]
Updated by