Project

General

Profile

Actions

Bug #13014

open

Charon.vici can get in a bad state

Added by Kris Phillips 3 months ago. Updated about 14 hours ago.

Status:
New
Priority:
Normal
Assignee:
Category:
IPsec
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
22.11
Release Notes:
Default
Affected Version:
2.6.0
Affected Architecture:
All

Description

The charon.vici daemon can get in a bad state where all of the qlen slots are "hung". This causes the Status --> IPSec and other webConfigurator elements to not properly display status. This does not affect the actual tunnels in any way and those will continue to pass traffic, but you cannot restart any of the tunnels, manually disconnect or connect them, restart the IPSec service, view the connected status of any Phase 1 or 2 tunnels, etc.

When this happens you will start seeing the following in the System Logs:
kernelsonewconn: pcb 0xfffff8011994b700: Listen queue overflow: 5 already in queue awaiting acceptance (1 occurrences)

You will also see this in netstat:

Current listen queue sizes (qlen/incqlen/maxqlen)
unix 5/0/3 /var/run/charon.vici


Files

kdump.JPG (43.2 KB) kdump.JPG Tobias Ock, 05/04/2022 03:29 AM

Related issues

Related to Bug #7420: ipsec status freezingNew03/23/2017

Actions
Actions #1

Updated by Jim Pingle 3 months ago

Might be the same root cause as #7420 though we don't have enough information about either one of these to say for certain. The symptoms are very similar, though.

I can't replicate this on demand, I've seen it happen maybe once or twice ever when working on IPsec code. There are a couple reports on the forum and reddit as well.

As far as I know, though, nobody can induce it reliably.

Actions #2

Updated by Jim Pingle 3 months ago

  • Related to Bug #7420: ipsec status freezing added
Actions #3

Updated by Pierre-Emmanuel DEGRYSE 3 months ago

Hi. I get the same error.

See below the IPSecs logs with the highest verbosity level:

Last 500 IPsec Log Entries. (Maximum 500)
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:56 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:56 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:56 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:55 charon 23603 03[CFG] vici client 36 disconnected
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 26 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 26 ready to write
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for writing
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for writing
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 01[CFG] vici client 36 requests: list-sas
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 26 ready to read
Apr 11 09:48:55 charon 23603 01[CFG] vici client 36 connected
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 26 ready to write
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for writing
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for writing
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 04[CFG] vici client 36 registered for: list-sa
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 26 ready to read
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 7 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 26 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:55 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:55 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:55 charon 23603 07[JOB] watched FD 24 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 5 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watched FD 18 ready to read
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watcher got notification, rebuilding
Apr 11 09:48:49 charon 23603 03[CFG] vici client 35 disconnected
Apr 11 09:48:49 charon 23603 07[JOB] watcher going to poll() 6 fds
Apr 11 09:48:49 charon 23603 07[JOB] watching 24 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 23 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 18 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 13 for reading
Apr 11 09:48:49 charon 23603 07[JOB] watching 8 for reading

Actions #4

Updated by Kris Phillips 2 months ago

Someone with this issue:

If you could please run:

ps aux | grep charon
Output should look something like this, with the bold number being important: root 35176 0.0 0.2 68960 19560 - I Thu21 0:06.64 /usr/local/libexec/ipsec/charon --use-syslog

ktrace -p [the PID of the charon process here from the above command]

kdump

Then please provide the output here in a redmine response. That will be most helpful in investigating this issue and resolving it.

Actions #5

Updated by Tobias Ock about 2 months ago

Hi,

after updating to pfSense Plus 22.01 on XG-7100 I get this issue too.
As a side note... We also changed the ports from 1Gb to 10Gb in this step

Unfortunately, kdump doesn't give me any output

Actions #6

Updated by Kris Phillips about 2 months ago

FYI, this seems to help: if you go to System --> Advanced --> System Tunables and change kern.ipc.soacceptqueue to at least 512, then reboot, it seems to abate whatever condition is causing this.

Actions #7

Updated by Brad Davis about 1 month ago

  • Assignee set to Mateusz Guzik
  • Target version set to 2.7.0
  • Plus Target Version set to 22.05

We think this is fixed, but need additional testing to know for sure.

Actions #8

Updated by Mateusz Guzik about 1 month ago

No, this is not fixed. However, chances are excellent this is an old & known bug: use-after-free in key-related state in ipsec. I have a patch to fix that bit, but it was not committed yet as running with it runs into another bug.

The idea will be to boot a kernel with custom debug which will either confirm the suspicion is correct OR give a starting point for investigation if not.

Actions #9

Updated by Jim Pingle about 1 month ago

  • Plus Target Version changed from 22.05 to 22.09
Actions #10

Updated by Jim Pingle about 14 hours ago

  • Plus Target Version changed from 22.09 to 22.11
Actions

Also available in: Atom PDF