Project

General

Profile

Bug #9934

suricata update kills WAN interface

Added by Srijan Nandi 4 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Suricata
Target version:
-
Start date:
11/28/2019
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.4.4-p3
Affected Architecture:

Description

Hello Everyone,

I am running pfSense 2.4.4-RELEASE-p3 (amd64) with suricata VERSION 4.1.5_2. I had set suricata to update rules every 12 hours at the default time of 00:30. Each time suricata updates it's rule set, it sets the interface down and then it comes up after 15-20 seconds. As I am running suricata on the WAN interface, internet goes down till the time WAN comes back up again.

It happens twice in a day as the update interval is 12 hrs.

As a workaround, I have changed the update interval to 1 Day and set the time to 05:30, when traffic is very less.

Logs:

Nov 26 00:30:10 pfSense php-cgi: [Suricata] ERROR: Rules download error: Operation timed out after 10003 milliseconds with 0 out of 0 bytes received
Nov 26 00:30:10 pfSense php-cgi: [Suricata] Will retry the download in 15 seconds...
Nov 26 00:30:27 pfSense php-cgi: File 'emerging.rules.tar.gz.md5' download attempts: 2 ...
Nov 26 00:30:27 pfSense php-cgi: [Suricata] Emerging Threats Open rules are up to date...
Nov 26 00:30:27 pfSense php-cgi: [Suricata] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
Nov 26 00:30:35 pfSense php-cgi: [Suricata] Snort GPLv2 Community Rules file update downloaded successfully.
Nov 26 00:30:36 pfSense php-cgi: [Suricata] Updating rules configuration for: BLAZENET_ISP_1 ...
Nov 26 00:30:37 pfSense php-cgi: [Suricata] Building new sid-msg.map file for BLAZENET_ISP_1...
Nov 26 00:30:37 pfSense SuricataStartup69633: Suricata STOP for BLAZENET_ISP_1_WAN(27120_em0)...
Nov 26 00:30:38 pfSense kernel: em0: link state changed to DOWN
Nov 26 00:30:38 pfSense check_reload_status: Linkup starting em0
Nov 26 00:30:39 pfSense php-fpm: /rc.linkup: Hotplug event detected for BLAZENET_ISP_1(wan) static IP (X.X.X.X )
Nov 26 00:30:39 pfSense check_reload_status: Reloading filter
Nov 26 00:30:39 pfSense php-cgi: [Suricata] Suricata has restarted with your new set of rules...
Nov 26 00:30:39 pfSense php-cgi: [Suricata] The Rules update has finished.
Nov 26 00:30:39 pfSense SuricataStartup76330: Suricata START for BLAZENET_ISP_1_WAN(27120_em0)...

Nov 26 00:30:54 pfSense check_reload_status: Linkup starting em0
Nov 26 00:30:54 pfSense kernel: em0: link state changed to UP
Nov 26 00:30:55 pfSense php-fpm: /rc.linkup: Hotplug event detected for BLAZENET_ISP_1(wan) static IP (X.X.X.X )
Nov 26 00:30:55 pfSense check_reload_status: rc.newwanip starting em0
Nov 26 00:30:55 pfSense check_reload_status: Reloading filter
Nov 26 00:30:56 pfSense php-fpm: /rc.newwanip: rc.newwanip: Info: starting on em0.
Nov 26 00:30:56 pfSense php-fpm: /rc.newwanip: rc.newwanip: on (IP address: X.X.X.X) (interface: BLAZENET_ISP_1[wan]) (real interface: em0).
Nov 26 00:30:56 pfSense check_reload_status: Reloading filter

Nov 26 06:00:00 pfSense php-cgi: [Suricata] Checking for updated MaxMind GeoLite2 IP database file...
Nov 26 06:00:01 pfSense php-cgi: [Suricata] GeoLite2-Country IP database is up-to-date.
Nov 26 06:00:01 pfSense php-cgi: [Suricata] GeoLite2-Country database update check finished.
Nov 26 12:30:07 pfSense php-cgi: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Nov 26 12:30:19 pfSense php-cgi: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Nov 26 12:30:20 pfSense php-cgi: [Suricata] Snort GPLv2 Community Rules are up to date...
Nov 26 12:30:22 pfSense php-cgi: [Suricata] Updating rules configuration for: BLAZENET_ISP_1 ...
Nov 26 12:30:23 pfSense php-cgi: [Suricata] Building new sid-msg.map file for BLAZENET_ISP_1...
Nov 26 12:30:23 pfSense SuricataStartup32425: Suricata STOP for BLAZENET_ISP_1_WAN(27120_em0)...
Nov 26 12:30:25 pfSense kernel: em0: link state changed to DOWN
Nov 26 12:30:25 pfSense check_reload_status: Linkup starting em0
Nov 26 12:30:26 pfSense php-fpm: /rc.linkup: Hotplug event detected for BLAZENET_ISP_1(wan) static IP (X.X.X.X )
Nov 26 12:30:26 pfSense check_reload_status: Reloading filter
Nov 26 12:30:26 pfSense php-cgi: [Suricata] Suricata has restarted with your new set of rules...
Nov 26 12:30:26 pfSense php-cgi: [Suricata] The Rules update has finished.
Nov 26 12:30:26 pfSense SuricataStartup36834: Suricata START for BLAZENET_ISP_1_WAN(27120_em0)...
Nov 26 12:30:26 pfSense check_reload_status: Syncing firewall

Nov 26 12:30:44 pfSense check_reload_status: Linkup starting em0
Nov 26 12:30:44 pfSense kernel: em0: link state changed to UP
Nov 26 12:30:45 pfSense php-fpm: /rc.linkup: Hotplug event detected for BLAZENET_ISP_1(wan) static IP (X.X.X.X )
Nov 26 12:30:45 pfSense check_reload_status: rc.newwanip starting em0
Nov 26 12:30:45 pfSense check_reload_status: Reloading filter
Nov 26 12:30:46 pfSense php-fpm: /rc.newwanip: rc.newwanip: Info: starting on em0.
Nov 26 12:30:46 pfSense php-fpm: /rc.newwanip: rc.newwanip: on (IP address: X.X.X.X) (interface: BLAZENET_ISP_1[wan]) (real interface: em0).
Nov 26 12:30:46 pfSense check_reload_status: Reloading filter

suricata.yaml (11.8 KB) suricata.yaml XG-1537 ix1 suricata.yaml Viktor Gurov, 02/07/2020 04:48 AM

History

#1 Updated by Srijan Nandi 4 months ago

Suricata is running in INLINE IPS mode. Every time, when suricata is stopped or started, it does a link up/down. Is this normal behaviour with INLINE IPS mode? Can it be modified to not restart interfaces while suricata restarts.

#2 Updated by Jim Pingle 4 months ago

  • Project changed from pfSense to pfSense Packages
  • Category changed from Package System to Suricata

#3 Updated by Danilo Zrenjanin 2 months ago

You can set Suricata to "Live Reload" the new rules without restarting itself.

Enable this global option to prevent a hard restart of Suricata when the rules reload:

Suricate > Global Settings, Live Rule Swap on Update.

#4 Updated by Bill Meeks 2 months ago

Suricata running with Inline IPS Mode uses the netmap kernel device. When Suricata stops and restarts, that also stops and restarts the netmap kernel device resulting in the network interface cycling. This is not a bug. It's just the way netmap works.

You have two options: (1) abandon Inline IPS Mode and use Legacy Mode with libpcap; or (2) do what user @Danilo Zrenjanin suggested and enable the "Live Reload" option for Suricata rules updates on the GLOBAL SETTINGS tab. This will not physically stop and restart Suricata on rules updates but will instead signal the running daemon of the need to reload the rules.

Note that choosing option #2 above may somewhat increase temporary use of RAM as for a period of time Suricata needs to keep two complete copies of the enabled rules in memory.

#5 Updated by Jim Pingle 2 months ago

  • Status changed from New to Not a Bug

#6 Updated by Viktor Gurov about 2 months ago

same issue on XG-1537 (pfSense 2.4.4-p3, suricata 4.1.6_3) with ix interface,
I found that killing suricata process (service stop) changes interface status to down ('no carrier' on the interface status page):

carp: 1@ix1: MASTER -> INIT (hardware interface down)
kernel: carp: demoted by 240 to 240 (interface down)
kernel: ix1: link state changed to DOWN

by "pkill -TERM -F /var/run/suricata_ix11234.pid" or "pkill -TERM -fn "suricata -i ix1"
but not in 100% of cases - sometime only 3rd or 4th restart 'down' the interface

also reproduced on APU.3B4 (2.4.5-RC, amd64) - but the interface did not remain in a “no carrier” state

SuricataStartup     45332     Suricata START for LAN(34549_igb0)...
Squid_Alarm     46079     Squid is disabled, exiting.
radiusd     35803     Ready to process requests
php-fpm     18547     /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
check_reload_status         Linkup starting igb0
kernel         igb0: link state changed to DOWN
kernel         igb0.33: link state changed to DOWN
kernel         igb0.99: link state changed to DOWN
kernel         igb0.111: link state changed to DOWN

<ips_mode>ips_mode_legacy</ips_mode> - in all tests

No such issue on VM or SG-1100 - but interfaces in both cases never go down

#7 Updated by Bill Meeks about 2 months ago

If Suricata is running using Legacy Mode Blocking, then the libpcap library is used and bonded to the interface where Suricata is enabled. The custom blocking module is not involved in this process at all. The standard pcap-binding process is used within the Suricata binary. Perhaps something has changed in that code from upstream, or something has changed in the libpcap library ???

I can look through the commits for Suricata upstream to see if anything has been changed lately in the PCAP portion of the code.

#8 Updated by Bill Meeks about 2 months ago

A look through the Suricata source code shows that the Suricata binary, when running in PCAP mode, will send explicit commands to the OS to disable checksum offloading on the physical interface. Perhaps that is now causing the OS kernel to cycle the physical interface ???

It appears the Suricata binary will first get the current operating flags for the physical interface, and if checksum offloading is enabled it will be forcibly disabled. Later, when Suricata stops, the binary will restore the previous operating flags for the physical interface.

So how about repeating the above tests but with the checksum offloading and TCP segmentation offloading disabled within pfSense? That way Suricata will not attempt to change the offloading as it will be already correctly set.

Also available in: Atom PDF