Project

General

Profile

Actions

Bug #7420

open

ipsec status freezing

Added by Brice Figureau about 7 years ago. Updated about 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
IPsec
Target version:
-
Start date:
03/23/2017
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
2.4.4-p3
Affected Architecture:

Description

I upgraded a SG-8860 yesterday from 2.3.2_1 to 2.3.3_1 after I applied the bios upgrade.

Unfortunately, now the IPsec status page freezes everytime I navigate to it, same for the shell command ipsec status, exactly like bug #5520 back during 2.2.x.
Note that the tunnels also keep dropping (it is probably related), and I have to kill -9 the charon process for them to work again which is a PITA.

Let me know if you need more information.


Files

ipsecleases-truss.log (8.41 KB) ipsecleases-truss.log Viktor Gurov, 02/05/2020 11:50 PM
swanctl-truss.log (99.3 KB) swanctl-truss.log Viktor Gurov, 02/05/2020 11:50 PM

Related issues

Related to Bug #13014: Deadlock in Charon VICI interfaceResolvedKristof Provost

Actions
Actions #1

Updated by Mitch Claborn over 6 years ago

I am also having this problem with 2.4.2.

Command line doesn't help either:

swanctl --list-sas
connecting to 'unix:///var/run/charon.vici' failed: Connection refused
Error: connecting to 'default' URI failed: Connection refused
strongSwan 5.6.0 swanctl

Actions #2

Updated by Jim Pingle over 4 years ago

  • Status changed from New to Closed

Old report and no recent recurrences. Lots of things in this area have changed, so most likely it's either fixed no longer relevant.

Actions #3

Updated by Viktor Gurov about 4 years ago

  • Status changed from Closed to New
  • Affected Version changed from 2.3.3_1 to 2.4.4-p3

The same problem was detected on 2.4.4-p3 after upgrading from 2.4.4-p1 in one ticket:

# swanctl --list-sas
connecting to 'unix:///var/run/charon.vici' failed: Connection refused
Error: connecting to 'default' URI failed: Connection refused

after reinstalling pfSense The Webconfigurator still crashes if the Widget "IPsec status" is active.

it seems that the problem is in the pfSense_ipsec_list_sa() function, which is used in ipsec_list_sa() by the widget and status_ipsec.php

Actions #4

Updated by Viktor Gurov about 4 years ago

status_ipsec.php and widget use these functions:

ipsec_dump_spd(); - # /sbin/setkey -DP
ipsec_dump_sad(); - # /sbin/setkey -D
ipsec_dump_mobile(); - # /usr/local/sbin/ipsec leases
ipsec_list_sa(); - pfSense_ipsec_list_sa() (use /usr/local/sbin/swanctl --list-sas (?))

setkey -DP and setkey -D show the correct output, but two other commands:

/usr/local/sbin/ipsec leases
connecting to 'unix:///var/run/charon.ctl' failed: Connection refused
failed to connect to stroke socket 'unix:///var/run/charon.ctl'
/usr/local/sbin/swanctl --list-sas

connecting to 'unix:///var/run/charon.vici' failed: Connection refused
Error: connecting to 'default' URI failed: Connection refused
strongSwan 5.7.1 swanctl
usage:
swanctl --list-sas [--raw|--pretty]

similar issues:
https://wiki.strongswan.org/issues/1484 - It was all due to insufficient "Max open files" limit for the user used to run charon.
https://wiki.strongswan.org/issues/3294 - The # character is used for comments (the rest of the line is ignored). To use it in a config value (e.g. a secret) enclose the whole thing in double quotes

Actions #5

Updated by Viktor Gurov about 4 years ago

this is CE version of 2.4.4-p3:
FreeBSD pfsense28.bvt.de 11.2-RELEASE-p10 FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019 :/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense amd64

"Max open files" limit is enough:
kern.maxfiles: 258180
kern.maxfilesperproc: 232362

attached output from 'truss /usr/local/sbin/swanctl --list-sas > swanctl-truss.log' and 'truss /usr/local/sbin/ipsec leases > ipsecleases-truss.log'

Actions #6

Updated by Jim Pingle about 4 years ago

You are chasing the wrong end of the problem. We know that it can't talk to the VICI socket. What we do not know is why there are other processes stuck talking to the VICI socket before it reaches that point. Something is consuming all the available sockets and is stuck waiting there.

Actions #7

Updated by Viktor Gurov about 4 years ago

it works OK if you disable IPsec Mobile

"Several of the threads from the thread pool are "reserved" for long running tasks in internal modules and plugins." (strongswan)
https://wiki.strongswan.org/projects/strongswan/wiki/JobPriority

plugins part of the strongswan.conf:

plugins {
        # Load defaults
        include /var/etc/ipsec/strongswan.d/charon/*.conf        
                stroke {
            secrets_file = /var/etc/ipsec/ipsec.secrets
        }        unity {
            load = no
        }
        attr {
            subnet = x.x.x.x/y
            split-include = x.x.x.x/y
            # Search domain and default domain
            28674 = "xxx.int" 
            28675 = "xxx.int" 
            28672 = "Welcome to xxx.int" 
        }
        xauth-generic {
            script = /etc/inc/ipsec.auth-user.php
            authcfg = AD
        }    
}

Actions #8

Updated by Jim Pingle about 4 years ago

That doesn't quite line up because strongSwan may have many worker threads running, but it still only has a few open spots for the vici socket.

Consider this on 2.5.0:

: swanctl --stats | grep threads
worker threads: 16 total, 10 idle, working: 5/0/1/0
: netstat -Lan 
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address  
[...]
unix  0/0/3                            /var/run/charon.vici

16 threads, 10 idle, but only 3 total sockets in the listen queue for VICI.

Actions #9

Updated by Jim Pingle almost 2 years ago

  • Related to Bug #13014: Deadlock in Charon VICI interface added
Actions

Also available in: Atom PDF