Project

General

Profile

Actions

Regression #12045

closed

High CPU usage and slowness with ``pfctl -ss``

Added by Jim Pingle 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Urgent
Category:
Operating System
Target version:
Start date:
06/15/2021
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Force Exclusion
Affected Version:
Affected Architecture:

Description

Some users have found that pfctl -ss is taking consuming large amounts of CPU and taking much longer than it should to output data on 2.5.2-RC

https://forum.netgate.com/topic/164442/pftop-hangs-my-gui-in-2-5-2-rc

pfctl -si | grep -A4 State
State Table                          Total             Rate
  current entries                     2407
  searches                          404293          546.3/s
  inserts                            20698           28.0/s
  removals                           18291           24.7/s

time pfctl -ss
0.141u 140.191s 2:20.52 99.8%   203+177k 0+0io 0pf+0w

time pfctl -vvss
0.157u 87.638s 1:28.08 99.6%    203+177k 0+0io 0pf+0w

The delay and high CPU is causing a process backlog due to RRD, and can also cause pftop to fail (from the console or Diag > pftop)

ps -ax | grep pfctl
8530 - L 42:21.31 /sbin/pfctl -i ovpnc3 -Fs
17907 - I 0:00.00 sh -c /sbin/pfctl -vvss | /usr/bin/grep creator | /usr/bin/cut -d" " -f7 | /usr/bin/sort -u
18015 - R 337:49.40 /sbin/pfctl -vvss
38322 - RN 365:07.87 /sbin/pfctl -ss
61701 - I 0:00.00 sh -c /sbin/pfctl -vvss | /usr/bin/grep creator | /usr/bin/cut -d" " -f7 | /usr/bin/sort -u
61787 - L 17:56.93 /sbin/pfctl -vvss
82409 - I 0:00.00 sh -c /sbin/pfctl -vvss | /usr/bin/grep creator | /usr/bin/cut -d" " -f7 | /usr/bin/sort -u
82711 - L 198:37.49 /sbin/pfctl -vvss
84187 - RN 385:23.46 /sbin/pfctl -ss
95032 - I 0:00.00 sh -c /sbin/pfctl -vvss | /usr/bin/grep creator | /usr/bin/cut -d" " -f7 | /usr/bin/sort -u
95080 - R 281:20.27 /sbin/pfctl -vvss
43823 0 S+ 0:00.00 grep pfctl

ps -ax | grep pftop
11621 - L 11:49.66 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
13020 - L 13:16.32 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
17041 - L 12:59.58 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
39212 - L 14:19.03 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
39821 - R 11:33.73 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
40097 - L 15:23.33 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
40726 - L 15:38.67 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
40967 - L 13:05.67 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
68675 - L 7:31.30 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
71010 - L 13:15.83 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
79096 - L 11:04.39 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
85169 - L 11:26.45 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
87241 - L 13:15.88 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
29040 0 S+ 0:00.00 grep pftop
77243 1- L+ 0:22.80 pftop
10237 2- L+ 0:40.86 /usr/local/sbin/pftop
  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
18015 root          1  20    0   503M    61M CPU7     7 345:55  99.96% pfctl
38322 root          1  20   20   134M    18M CPU4     4 373:08  99.95% pfctl
84187 root          1  20   20   134M    18M CPU3     3 393:25  99.91% pfctl
95080 root          1  80    0   503M    61M RUN      5 289:22  99.90% pfctl
39212 root          1  20    0   112M    91M CPU6     6  18:23  62.80% pftop
39821 root          1  80    0   112M    92M *pf_id   4  15:57  38.04% pftop
30353 root          1  72    0   146M  3636K RUN      7   2:32   2.41% pftop
11621 root          1  20    0   113M    98M *pf_id   5  15:43   2.41% pftop
40726 root          1  75    0   112M  3264K CPU1     1  18:52   1.00% pftop
82711 root          1  81    0   503M    61M CPU0     0 199:33   0.95% pfctl

Kristof noted that it may be due to an nvlist bottleneck which was just fixed in FreeBSD:
https://cgit.freebsd.org/src/commit/?id=89d5cbb82294c8624e66f920d50353057ccab14b


Related issues

Related to Regression #12057: 21.09/2.6.0 - High CPU usage and slowness with ``pfctl -ss``FeedbackKristof Provost06/18/2021

Actions
Related to Regression #12069: Panic in ``pfctl`` with large numbers of statesFeedbackKristof Provost06/22/2021

Actions
Actions #1

Updated by Renato Botelho 3 months ago

  • Status changed from New to Feedback

I've cherry-picked commits from upstream/main to pfsense/RELENG_2_5_2 that should help this case:

b5d787d93b3d83f28e87e1f8cc740cb160f8f0ac
0020c845a086766b3315372f006363f8ad76ac54
d97753b5c8f1d32fbcdcbb0d129b49f808245865
3bea7b5b05f200df4cabee12e405b8feade16f0e
89d5cbb82294c8624e66f920d50353057ccab14b

Actions #2

Updated by Jim Pingle 3 months ago

  • Status changed from Feedback to Resolved

The latest build includes the fixes for this and it's working properly now. Dumping the states is fast no matter how many there are. Additional confirmation from the forum confirms it's working for users who saw problems before.

Actions #4

Updated by Jim Pingle 3 months ago

Yes, we are aware, but 2.6.0 will get the fix when we do a full sync with FreeBSD sources next, which wasn't an option for 2.5.2 (so we picked only the necessary commits there).

Actions #5

Updated by Jim Pingle 3 months ago

  • Related to Regression #12057: 21.09/2.6.0 - High CPU usage and slowness with ``pfctl -ss`` added
Actions #6

Updated by Jim Pingle 3 months ago

  • Related to Regression #12069: Panic in ``pfctl`` with large numbers of states added
Actions

Also available in: Atom PDF