Bug #7420
open
ipsec status freezing
Added by Brice Figureau about 7 years ago. Updated about 4 years ago.
0%
Description
I upgraded a SG-8860 yesterday from 2.3.2_1 to 2.3.3_1 after I applied the bios upgrade.
Unfortunately, now the IPsec status page freezes everytime I navigate to it, same for the shell command ipsec status, exactly like bug #5520 back during 2.2.x.
Note that the tunnels also keep dropping (it is probably related), and I have to kill -9 the charon process for them to work again which is a PITA.
Let me know if you need more information.
Files
| ipsecleases-truss.log (8.41 KB) ipsecleases-truss.log | Viktor Gurov, 02/05/2020 11:50 PM | ||
| swanctl-truss.log (99.3 KB) swanctl-truss.log | Viktor Gurov, 02/05/2020 11:50 PM |
Related issues
Updated by Mitch Claborn over 6 years ago
I am also having this problem with 2.4.2.
Command line doesn't help either:
swanctl --list-sas
connecting to 'unix:///var/run/charon.vici' failed: Connection refused
Error: connecting to 'default' URI failed: Connection refused
strongSwan 5.6.0 swanctl
Updated by Jim Pingle over 4 years ago
- Status changed from New to Closed
Old report and no recent recurrences. Lots of things in this area have changed, so most likely it's either fixed no longer relevant.
Updated by Viktor Gurov about 4 years ago
- Status changed from Closed to New
- Affected Version changed from 2.3.3_1 to 2.4.4-p3
The same problem was detected on 2.4.4-p3 after upgrading from 2.4.4-p1 in one ticket:
# swanctl --list-sas connecting to 'unix:///var/run/charon.vici' failed: Connection refused Error: connecting to 'default' URI failed: Connection refused
after reinstalling pfSense The Webconfigurator still crashes if the Widget "IPsec status" is active.
it seems that the problem is in the pfSense_ipsec_list_sa() function, which is used in ipsec_list_sa() by the widget and status_ipsec.php
Updated by Viktor Gurov about 4 years ago
status_ipsec.php and widget use these functions:
ipsec_dump_spd(); - # /sbin/setkey -DP
ipsec_dump_sad(); - # /sbin/setkey -D
ipsec_dump_mobile(); - # /usr/local/sbin/ipsec leases
ipsec_list_sa(); - pfSense_ipsec_list_sa() (use /usr/local/sbin/swanctl --list-sas (?))
setkey -DP and setkey -D show the correct output, but two other commands:
/usr/local/sbin/ipsec leases connecting to 'unix:///var/run/charon.ctl' failed: Connection refused failed to connect to stroke socket 'unix:///var/run/charon.ctl' /usr/local/sbin/swanctl --list-sas connecting to 'unix:///var/run/charon.vici' failed: Connection refused Error: connecting to 'default' URI failed: Connection refused strongSwan 5.7.1 swanctl usage: swanctl --list-sas [--raw|--pretty]
similar issues:
https://wiki.strongswan.org/issues/1484 - It was all due to insufficient "Max open files" limit for the user used to run charon.
https://wiki.strongswan.org/issues/3294 - The # character is used for comments (the rest of the line is ignored). To use it in a config value (e.g. a secret) enclose the whole thing in double quotes
Updated by Viktor Gurov about 4 years ago
- File ipsecleases-truss.log ipsecleases-truss.log added
- File swanctl-truss.log swanctl-truss.log added
this is CE version of 2.4.4-p3:
FreeBSD pfsense28.bvt.de 11.2-RELEASE-p10 FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019 root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense amd64
"Max open files" limit is enough:
kern.maxfiles: 258180
kern.maxfilesperproc: 232362
attached output from 'truss /usr/local/sbin/swanctl --list-sas > swanctl-truss.log' and 'truss /usr/local/sbin/ipsec leases > ipsecleases-truss.log'
Updated by Jim Pingle about 4 years ago
You are chasing the wrong end of the problem. We know that it can't talk to the VICI socket. What we do not know is why there are other processes stuck talking to the VICI socket before it reaches that point. Something is consuming all the available sockets and is stuck waiting there.
Updated by Viktor Gurov about 4 years ago
it works OK if you disable IPsec Mobile
"Several of the threads from the thread pool are "reserved" for long running tasks in internal modules and plugins." (strongswan)
https://wiki.strongswan.org/projects/strongswan/wiki/JobPriority
plugins part of the strongswan.conf:
plugins {
# Load defaults
include /var/etc/ipsec/strongswan.d/charon/*.conf
stroke {
secrets_file = /var/etc/ipsec/ipsec.secrets
} unity {
load = no
}
attr {
subnet = x.x.x.x/y
split-include = x.x.x.x/y
# Search domain and default domain
28674 = "xxx.int"
28675 = "xxx.int"
28672 = "Welcome to xxx.int"
}
xauth-generic {
script = /etc/inc/ipsec.auth-user.php
authcfg = AD
}
}
Updated by Jim Pingle about 4 years ago
That doesn't quite line up because strongSwan may have many worker threads running, but it still only has a few open spots for the vici socket.
Consider this on 2.5.0:
: swanctl --stats | grep threads worker threads: 16 total, 10 idle, working: 5/0/1/0
: netstat -Lan Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address [...] unix 0/0/3 /var/run/charon.vici
16 threads, 10 idle, but only 3 total sockets in the listen queue for VICI.
Updated by Jim Pingle about 2 years ago
- Related to Bug #13014: Deadlock in Charon VICI interface added