Project

General

Profile

Bug #7420

ipsec status freezing

Added by Brice Figureau about 3 years ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
IPsec
Target version:
-
Start date:
03/23/2017
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.4.4-p3
Affected Architecture:

Description

I upgraded a SG-8860 yesterday from 2.3.2_1 to 2.3.3_1 after I applied the bios upgrade.

Unfortunately, now the IPsec status page freezes everytime I navigate to it, same for the shell command ipsec status, exactly like bug #5520 back during 2.2.x.
Note that the tunnels also keep dropping (it is probably related), and I have to kill -9 the charon process for them to work again which is a PITA.

Let me know if you need more information.

ipsecleases-truss.log (8.41 KB) ipsecleases-truss.log Viktor Gurov, 02/05/2020 11:50 PM
swanctl-truss.log (99.3 KB) swanctl-truss.log Viktor Gurov, 02/05/2020 11:50 PM

History

#1 Updated by Mitch Claborn over 2 years ago

I am also having this problem with 2.4.2.

Command line doesn't help either:

swanctl --list-sas
connecting to 'unix:///var/run/charon.vici' failed: Connection refused
Error: connecting to 'default' URI failed: Connection refused
strongSwan 5.6.0 swanctl

#2 Updated by Jim Pingle 7 months ago

  • Status changed from New to Closed

Old report and no recent recurrences. Lots of things in this area have changed, so most likely it's either fixed no longer relevant.

#3 Updated by Viktor Gurov about 2 months ago

  • Status changed from Closed to New
  • Affected Version changed from 2.3.3_1 to 2.4.4-p3

The same problem was detected on 2.4.4-p3 after upgrading from 2.4.4-p1 in one ticket:

# swanctl --list-sas
connecting to 'unix:///var/run/charon.vici' failed: Connection refused
Error: connecting to 'default' URI failed: Connection refused

after reinstalling pfSense The Webconfigurator still crashes if the Widget "IPsec status" is active.

it seems that the problem is in the pfSense_ipsec_list_sa() function, which is used in ipsec_list_sa() by the widget and status_ipsec.php

#4 Updated by Viktor Gurov about 2 months ago

status_ipsec.php and widget use these functions:

ipsec_dump_spd(); - # /sbin/setkey -DP
ipsec_dump_sad(); - # /sbin/setkey -D
ipsec_dump_mobile(); - # /usr/local/sbin/ipsec leases
ipsec_list_sa(); - pfSense_ipsec_list_sa() (use /usr/local/sbin/swanctl --list-sas (?))

setkey -DP and setkey -D show the correct output, but two other commands:

/usr/local/sbin/ipsec leases
connecting to 'unix:///var/run/charon.ctl' failed: Connection refused
failed to connect to stroke socket 'unix:///var/run/charon.ctl'
/usr/local/sbin/swanctl --list-sas

connecting to 'unix:///var/run/charon.vici' failed: Connection refused
Error: connecting to 'default' URI failed: Connection refused
strongSwan 5.7.1 swanctl
usage:
swanctl --list-sas [--raw|--pretty]

similar issues:
https://wiki.strongswan.org/issues/1484 - It was all due to insufficient "Max open files" limit for the user used to run charon.
https://wiki.strongswan.org/issues/3294 - The # character is used for comments (the rest of the line is ignored). To use it in a config value (e.g. a secret) enclose the whole thing in double quotes

#5 Updated by Viktor Gurov about 2 months ago

this is CE version of 2.4.4-p3:
FreeBSD pfsense28.bvt.de 11.2-RELEASE-p10 FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019 :/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense amd64

"Max open files" limit is enough:
kern.maxfiles: 258180
kern.maxfilesperproc: 232362

attached output from 'truss /usr/local/sbin/swanctl --list-sas > swanctl-truss.log' and 'truss /usr/local/sbin/ipsec leases > ipsecleases-truss.log'

#6 Updated by Jim Pingle about 2 months ago

You are chasing the wrong end of the problem. We know that it can't talk to the VICI socket. What we do not know is why there are other processes stuck talking to the VICI socket before it reaches that point. Something is consuming all the available sockets and is stuck waiting there.

#7 Updated by Viktor Gurov about 1 month ago

it works OK if you disable IPsec Mobile

"Several of the threads from the thread pool are "reserved" for long running tasks in internal modules and plugins." (strongswan)
https://wiki.strongswan.org/projects/strongswan/wiki/JobPriority

plugins part of the strongswan.conf:

plugins {
        # Load defaults
        include /var/etc/ipsec/strongswan.d/charon/*.conf        
                stroke {
            secrets_file = /var/etc/ipsec/ipsec.secrets
        }        unity {
            load = no
        }
        attr {
            subnet = x.x.x.x/y
            split-include = x.x.x.x/y
            # Search domain and default domain
            28674 = "xxx.int" 
            28675 = "xxx.int" 
            28672 = "Welcome to xxx.int" 
        }
        xauth-generic {
            script = /etc/inc/ipsec.auth-user.php
            authcfg = AD
        }    
}

#8 Updated by Jim Pingle about 1 month ago

That doesn't quite line up because strongSwan may have many worker threads running, but it still only has a few open spots for the vici socket.

Consider this on 2.5.0:

: swanctl --stats | grep threads
worker threads: 16 total, 10 idle, working: 5/0/1/0
: netstat -Lan 
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address  
[...]
unix  0/0/3                            /var/run/charon.vici

16 threads, 10 idle, but only 3 total sockets in the listen queue for VICI.

Also available in: Atom PDF