Bug #15516: Per-rule byte counter values lost across a filter reload - pfSense - pfSense bugtracker

Actions

Copy link

Bug #15516

closed

Per-rule byte counter values lost across a filter reload

Added by Steve Wheeler 6 months ago. Updated about 1 month ago.

Status:

Resolved

Priority:

Normal

Assignee:

Mateusz Guzik

Category:

Rules / NAT

Target version:

2.8.0

Start date:

Due date:

% Done:

Estimated time:

Plus Target Version:

24.11

Release Notes:

Default

Affected Version:

2.7.2

Affected Architecture:

Description

In some situations the byte counter reported by pfctl is not retained correctly across a filter reload.

It appears to be dependent on the value stored. So 4414450379 Bytes is retained correctly but 4793436239 Bytes is not:

[24.03-RELEASE][admin@4200.stevew.lan]/root: pfctl -vvsr | grep -A3 @90
@90 pass in quick on igc3 reply-to (igc3 172.21.16.1) inet proto tcp from <WAN__NETWORK:1> to 172.21.16.11 port = 5201 flags S/SA keep state (if-bound) label "USER_RULE: Allow iperf" label "id:1716563048" ridentifier 1716563048
  [ Evaluations: 95        Packets: 6177197   Bytes: 4793436239  States: 2     ]
  [ Inserted: uid 0 pid 0 State Creations: 2     ]
  [ Last Active Time: Fri May 24 21:27:52 2024 ]
[24.03-RELEASE][admin@4200.stevew.lan]/root: /etc/rc.filter_configure
[24.03-RELEASE][admin@4200.stevew.lan]/root: pfctl -vvsr | grep -A3 @90
@90 pass in quick on igc3 reply-to (igc3 172.21.16.1) inet proto tcp from <WAN__NETWORK:1> to 172.21.16.11 port = 5201 flags S/SA keep state (if-bound) label "USER_RULE: Allow iperf" label "id:1716563048" ridentifier 1716563048
  [ Evaluations: 96        Packets: 6177197   Bytes: 498468943   States: 0     ]
  [ Inserted: uid 0 pid 0 State Creations: 0     ]
  [ Last Active Time: N/A ]

The packet count remains the same.

Tested in 24.03 amd64

It does not happen if you then pass more traffic raising the byte counter above the limit and then reload the filter a second time.

Actions

Copy link

Updated by Steve Y 6 months ago

Just to add another data point, as I mentioned in https://forum.netgate.com/topic/188336/firewall-rule-counters-max-size/17 a counter of 96 GB was reset but a few at 5+ GB were not.

Actions

Copy link

Updated by Jim Pingle 6 months ago

Plus Target Version changed from 24.07 to 24.08

Actions

Copy link

Updated by Kristof Provost 6 months ago

I believe I see what's happening.

When rules are re-loaded we explicitly copy the counter state from the previous rules to the new rules. This uses the pf_counter_u64_add_protected() function (specifically introduced to work around counter_64 performance issues on 32-bit platforms, i.e. 3100), but this function takes a uint32_t as an argument.
So whenever our counter is over 32-bits we end up losing the upper bits.

This is masked to some extent because the internal counters are separated between in- and out-bound counters, but those get added up in userspace before they're displayed to users.

Actions

Copy link

Updated by Mateusz Guzik 6 months ago

Assignee set to Mateusz Guzik

I agree with the analysis.

The entire thing however showed up because of performance problems on the 3100.

It very much can be addressed with modest hassle while retaining everything, which I'm going to sort out later today or tomorrow.

Actions

Copy link

Updated by Mateusz Guzik 5 months ago

Status changed from New to Feedback

Fixed in https://gitlab.netgate.com/pfSense/FreeBSD-src/-/commit/5c71199c72c42a9836ed6f36521a7ff95e169e0b

Steve and Kristof verified the 3100 and amd64 respectively.

Actions

Copy link

Updated by Steve Wheeler 5 months ago

Looks good with the patched kernel:

[24.08-DEVELOPMENT][admin@3100.stevew.lan]/root: pfctl -vvsr | grep -A3 @88
@88 pass in quick on mvneta1.4084 reply-to (mvneta1.4084 172.21.16.1) inet all flags S/SA keep state (if-bound) label "USER_RULE: Allow all ipv4+ipv6 via pfSsh.php" label "id:1652894570" ridentifier 1652894570
  [ Evaluations: 360       Packets: 7086185   Bytes: 5491882806  States: 7     ]
  [ Inserted: uid 0 pid 0 State Creations: 11    ]
  [ Last Active Time: Thu Jun 20 17:48:03 2024 ]
[24.08-DEVELOPMENT][admin@3100.stevew.lan]/root: /etc/rc.filter_configure
[24.08-DEVELOPMENT][admin@3100.stevew.lan]/root: pfctl -vvsr | grep -A3 @88
@88 pass in quick on mvneta1.4084 reply-to (mvneta1.4084 172.21.16.1) inet all flags S/SA keep state (if-bound) label "USER_RULE: Allow all ipv4+ipv6 via pfSsh.php" label "id:1652894570" ridentifier 1652894570
  [ Evaluations: 366       Packets: 7086207   Bytes: 5491885050  States: 0     ]
  [ Inserted: uid 0 pid 0 State Creations: 0     ]
  [ Last Active Time: N/A ]

Actions

Copy link