pfSense

2021/02/04 09:33:29 [error] 38147#100184: *3 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.101.30, server: , request: "GET /firewall_nat_out.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.101.254:2253", referrer: "https://192.168.101.254:2253/firewall_nat_out_edit.php?id=22"

I just ran into the same thing on one out of 3 lab machines.

Each one announces one /44 or /48, receives default route only and has about the same specs. Usual load very low. Load now 1.50 - 2.50, using more than 1GB of RAM and swapping out, web GUI timing out.

The only thing different on the instance that is locked up now is that it is on a different hypervisor. It is on VMware ESxi with VMware tools packages installed. I will try reach out to the original reporter to compare notes and see if we can come up with something tangible.

i have no install vmware, I installed the pfsense 2.5 system on the hardware server. AMD CPU and DDR3 16G RAM.SSD DISK

just i am running frr bgp, get a lot of full route tables from some upstream. Because I have many different locations and different servers upstream.

ihave tested frr use the "service integrated-vtysh-config" run in ubuntu system. This mode also takes up a lot of memory and resources. if deleted the "service integrated-vtysh-config" Will reduce the use of memory resources by at least twice

When FRR is stopped, resources will be relieved

M Felden wrote:

I just ran into the same thing on one out of 3 lab machines.

Each one announces one /44 or /48, receives default route only and has about the same specs. Usual load very low. Load now 1.50 - 2.50, using more than 1GB of RAM and swapping out, web GUI timing out.

The only thing different on the instance that is locked up now is that it is on a different hypervisor. It is on VMware ESxi with VMware tools packages installed. I will try reach out to the original reporter to compare notes and see if we can come up with something tangible.

when you try receives a lot of full routes,For example, millions and tens of millions of routes, You will see that the CPU is full,The entire pfsense webgui is locked.

vmstat 1 5
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id
7 0 0 5.2G 11G 44086 0 1 1 25627 666 0 0 1270 101607 28731 42 17 41
7 0 0 5.2G 11G 38568 0 0 0 12 1095 0 0 187 42791 20167 87 13 0
7 0 0 5.1G 11G 42805 0 0 0 27389 1107 0 2 181 34174 18188 92 8 0
7 0 0 5.0G 12G 37077 0 0 0 64914 1068 0 0 197 12521 5629 91 9 0
7 0 0 5.2G 11G 80071 0 0 0 31381 1095 0 0 160 21740 5395 49 47 4

ps aux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 88102 29.8 0.2 52392 39072 - R 09:49 0:02.17 /usr/bin/netstat --libxo json nWr
root 88401 28.0 0.2 52392 39072 - R 09:49 0:01.06 /usr/bin/netstat --libxo json -nWr
root 87946 27.9 0.0 0 16 - RE 09:49 0:02.11 /usr/bin/netstat --libxo json -nWr
root 88229 25.5 0.2 52392 39072 - R 09:49 0:01.40 /usr/bin/netstat --libxo json -nWr
root 62236 18.4 1.1 274200 184236 - S 09:14 4:17.16 php-fpm: pool nginx (php-fpm)
root 33221 18.1 1.2 281348 189380 - S 08:49 3:59.71 php-fpm: pool nginx (php-fpm)
root 84966 15.9 1.2 278220 186348 - S 09:25 3:19.80 php-fpm: pool nginx (php-fpm)
root 66697 15.5 1.2 277456 186684 - S 08:49 3:39.26 php-fpm: pool nginx (php-fpm)
root 43319 13.4 1.2 283168 193596 - S 08:49 6:13.60 php-fpm: pool nginx (php-fpm)
root 31932 11.8 1.2 289688 198684 - S 08:49 4:06.72 php-fpm: pool nginx (php-fpm)
root 89231 11.8 1.4 312764 219892 - R 08:49 7:34.97 php-fpm: pool nginx (php-fpm)
root 11 9.0 0.0 0 64 - RNL 08:49 79:14.65 [idle]
root 12 1.0 0.0 0 496 - WL 08:49 2:08.37 [intr]
frr 61216 0.1 0.4 168396 70232 - Ss 09:25 1:12.99 /usr/local/sbin/zebra -P0 -d
root 0 0.0 0.0 0 576 - DLs 08:49 0:41.21 [kernel]
root 1 0.0 0.0 9548 940 - ILs 08:49 0:00.03 /sbin/init -
root 2 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto]
root 3 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto returns 0]
root 4 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto returns 1]
root 5 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto returns 2]
root 6 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto returns 3]
root 7 0.0 0.0 0 32 - DL 08:49 0:04.57 [cam]
root 8 0.0 0.0 0 16 - DL 08:49 0:00.00 [soaiod1]
root 9 0.0 0.0 0 16 - DL 08:49 0:00.00 [soaiod2]
root 10 0.0 0.0 0 16 - DL 08:49 0:00.00 [audit]
root 13 0.0 0.0 0 64 - DL 08:49 0:00.01 [ng_queue]
root 14 0.0 0.0 0 48 - DL 08:49 0:00.01 [geom]
root 15 0.0 0.0 0 16 - DL 08:49 0:00.00 [sequencer 00]
root 16 0.0 0.0 0 640 - DL 08:49 0:04.57 [usb]
root 17 0.0 0.0 0 16 - DL 08:49 0:00.00 [soaiod3]
root 18 0.0 0.0 0 16 - DL 08:49 0:00.00 [soaiod4]
root 19 0.0 0.0 0 16 - DL 08:49 0:00.00 [sctp_iterator]
root 20 0.0 0.0 0 16 - DL 08:49 0:14.21 [pf purge]
root 21 0.0 0.0 0 16 - DL 08:49 0:01.16 [rand_harvestq]
root 22 0.0 0.0 0 48 - DL 08:49 0:00.87 [pagedaemon]
root 23 0.0 0.0 0 16 - DL 08:49 0:00.00 [vmdaemon]
root 24 0.0 0.0 0 144 - DL 08:49 0:00.38 [bufdaemon]
root 25 0.0 0.0 0 16 - DL 08:49 0:00.01 [vnlru]
root 26 0.0 0.0 0 16 - DL 08:49 0:00.28 [syncer]
root 27 0.0 0.0 0 16 - DL 08:49 0:00.00 [ALQ Daemon]
root 67 0.0 0.0 0 16 - DL 08:49 0:00.03 [md0]
root 781 0.0 0.2 104136 32880 - Ss 08:49 0:00.08 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root 820 0.0 0.0 11388 2948 - INs 08:49 0:00.03 /usr/local/sbin/check_reload_status
root 822 0.0 0.0 11340 2740 - IN 08:49 0:00.00 check_reload_status: Monitoring daemon of check_reload_status (check_reload_status)
root 990 0.0 0.0 10024 1484 - Ss 08:49 0:00.02 /sbin/devd -q -f /etc/pfSense-devd.conf
root 10106 0.0 0.1 20056 10820 - Ss 08:50 0:00.13 sshd: admin@pts/0 (sshd)
root 10706 0.0 0.3 137404 47276 - I 08:49 0:28.31 php-fpm: pool nginx (php-fpm)
root 11827 0.0 0.0 11392 2804 - Ss 08:50 0:00.25 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
root 12249 0.0 0.1 21300 9724 - Ss 08:49 0:00.04 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient
root 12266 0.0 0.1 19612 10512 - Is 08:49 0:00.00 /usr/sbin/sshd
root 12428 0.0 0.0 10692 2244 - Is 08:50 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root 12889 0.0 0.0 10716 2264 - I 08:50 0:00.00 minicron: helper /usr/local/bin/ping_hosts.sh (minicron)
root 12915 0.0 0.0 10692 2248 - Is 08:50 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
root 13099 0.0 0.0 10716 2268 - I 08:50 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts (minicron)
root 13406 0.0 0.0 10692 2248 - Is 08:50 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data
root 13618 0.0 0.0 10716 2268 - I 08:50 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data (minicron)
root 17906 0.0 0.1 21300 9744 - Ss 08:49 0:00.04 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt3.conf -p /var/run/pppoe_opt3.pid -s ppp pppoeclient
root 20903 0.0 0.1 21300 9744 - Ss 08:49 0:00.04 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt8.conf -p /var/run/pppoe_opt8.pid -s ppp pppoeclient
dhcpd 24683 0.0 0.1 23156 12056 - Ss 09:15 0:00.74 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid lagg0
root 24967 0.0 0.0 17404 8048 - Ss 08:51 0:01.32 /usr/local/sbin/openvpn --config /var/etc/openvpn/client9/config.ovpn
dhcpd 35020 0.0 0.1 25228 14424 - Ss 09:15 0:00.71 /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid lagg0
root 35428 0.0 0.0 11192 2612 - Is 08:49 0:00.01 /usr/local/sbin/dhcp6c -d -n -c /var/etc/dhcp6c_opt8.conf -p /var/run/dhcp6c_pppoe1.pid pppoe1
root 35763 0.0 0.0 10584 2212 - Is 09:15 0:00.00 /usr/local/sbin/dhcpleases6 -c /usr/local/bin/php-cgi -f /usr/local/sbin/prefixes.php -l /var/dhcpd/var/db/dhcpd6.leases
root 37352 0.0 0.0 11592 3104 - IN 09:15 0:00.86 /bin/sh /var/db/rrd/updaterrd.sh
root 42199 0.0 0.1 28108 8312 - Is 08:49 0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
root 42286 0.0 0.1 28636 8772 - I 08:49 0:00.00 nginx: worker process (nginx)
root 42571 0.0 0.1 30284 10092 - I 08:49 0:00.40 nginx: worker process (nginx)
root 42871 0.0 0.0 11356 2728 - Ss 08:49 0:00.02 /usr/sbin/cron -s
root 43071 0.0 0.0 17408 8048 - Ss 08:50 0:03.91 /usr/local/sbin/openvpn --config /var/etc/openvpn/client3/config.ovpn
root 43733 0.0 0.0 19252 7352 - Ss 08:49 1:49.04 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
root 51389 0.0 0.0 11172 2576 - Ss 08:49 0:00.83 /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
root 52427 0.0 0.1 20056 10820 - Ss 09:36 0:00.11 sshd: admin@pts/1 (sshd)
root 57658 0.0 0.0 11192 2612 - Is 08:49 0:00.05 /usr/local/sbin/dhcp6c -d -n -c /var/etc/dhcp6c_opt3.conf -p /var/run/dhcp6c_pppoe2.pid pppoe2
frr 62005 0.0 0.0 30584 6668 - Ss 09:25 0:00.03 /usr/local/sbin/staticd -d
frr 63150 0.0 7.5 1267584 1212368 - Ss 09:25 0:38.60 /usr/local/sbin/bgpd -M rpki -d
root 66958 0.0 0.0 11516 3056 - Is 09:36 0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root 67272 0.0 0.0 10624 2220 - I 09:36 0:00.00 /bin/cat
root 67288 0.0 0.0 16040 3988 - IC 09:36 0:00.00 /usr/local/libexec/sshg-parser
root 67597 0.0 0.0 11936 2744 - IC 09:36 0:00.00 /usr/local/libexec/sshg-blocker
root 67637 0.0 0.0 11516 3052 - I 09:36 0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root 67827 0.0 0.0 11500 3044 - I 09:36 0:00.00 /bin/sh /usr/local/libexec/sshg-fw-pf
root 70412 0.0 0.0 12188 3252 - Ss 08:49 0:00.08 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid

Turns out my issue was unrelated. My issue was https://redmine.pfsense.org/issues/11404

My instance was accepting more routes than what can fit in its RAM and things started falling over.