Bug #9934
closed
suricata update kills WAN interface
0%
Description
Hello Everyone,
I am running pfSense 2.4.4-RELEASE-p3 (amd64) with suricata VERSION 4.1.5_2. I had set suricata to update rules every 12 hours at the default time of 00:30. Each time suricata updates it's rule set, it sets the interface down and then it comes up after 15-20 seconds. As I am running suricata on the WAN interface, internet goes down till the time WAN comes back up again.
It happens twice in a day as the update interval is 12 hrs.
As a workaround, I have changed the update interval to 1 Day and set the time to 05:30, when traffic is very less.
Logs:
Nov 26 00:30:10 pfSense php-cgi: [Suricata] ERROR: Rules download error: Operation timed out after 10003 milliseconds with 0 out of 0 bytes received
Nov 26 00:30:10 pfSense php-cgi: [Suricata] Will retry the download in 15 seconds...
Nov 26 00:30:27 pfSense php-cgi: File 'emerging.rules.tar.gz.md5' download attempts: 2 ...
Nov 26 00:30:27 pfSense php-cgi: [Suricata] Emerging Threats Open rules are up to date...
Nov 26 00:30:27 pfSense php-cgi: [Suricata] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
Nov 26 00:30:35 pfSense php-cgi: [Suricata] Snort GPLv2 Community Rules file update downloaded successfully.
Nov 26 00:30:36 pfSense php-cgi: [Suricata] Updating rules configuration for: BLAZENET_ISP_1 ...
Nov 26 00:30:37 pfSense php-cgi: [Suricata] Building new sid-msg.map file for BLAZENET_ISP_1...
Nov 26 00:30:37 pfSense SuricataStartup69633: Suricata STOP for BLAZENET_ISP_1_WAN(27120_em0)...
Nov 26 00:30:38 pfSense kernel: em0: link state changed to DOWN
Nov 26 00:30:38 pfSense check_reload_status: Linkup starting em0
Nov 26 00:30:39 pfSense php-fpm: /rc.linkup: Hotplug event detected for BLAZENET_ISP_1(wan) static IP (X.X.X.X )
Nov 26 00:30:39 pfSense check_reload_status: Reloading filter
Nov 26 00:30:39 pfSense php-cgi: [Suricata] Suricata has restarted with your new set of rules...
Nov 26 00:30:39 pfSense php-cgi: [Suricata] The Rules update has finished.
Nov 26 00:30:39 pfSense SuricataStartup76330: Suricata START for BLAZENET_ISP_1_WAN(27120_em0)...
Nov 26 00:30:54 pfSense check_reload_status: Linkup starting em0
Nov 26 00:30:54 pfSense kernel: em0: link state changed to UP
Nov 26 00:30:55 pfSense php-fpm: /rc.linkup: Hotplug event detected for BLAZENET_ISP_1(wan) static IP (X.X.X.X )
Nov 26 00:30:55 pfSense check_reload_status: rc.newwanip starting em0
Nov 26 00:30:55 pfSense check_reload_status: Reloading filter
Nov 26 00:30:56 pfSense php-fpm: /rc.newwanip: rc.newwanip: Info: starting on em0.
Nov 26 00:30:56 pfSense php-fpm: /rc.newwanip: rc.newwanip: on (IP address: X.X.X.X) (interface: BLAZENET_ISP_1[wan]) (real interface: em0).
Nov 26 00:30:56 pfSense check_reload_status: Reloading filter
Nov 26 06:00:00 pfSense php-cgi: [Suricata] Checking for updated MaxMind GeoLite2 IP database file...
Nov 26 06:00:01 pfSense php-cgi: [Suricata] GeoLite2-Country IP database is up-to-date.
Nov 26 06:00:01 pfSense php-cgi: [Suricata] GeoLite2-Country database update check finished.
Nov 26 12:30:07 pfSense php-cgi: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Nov 26 12:30:19 pfSense php-cgi: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Nov 26 12:30:20 pfSense php-cgi: [Suricata] Snort GPLv2 Community Rules are up to date...
Nov 26 12:30:22 pfSense php-cgi: [Suricata] Updating rules configuration for: BLAZENET_ISP_1 ...
Nov 26 12:30:23 pfSense php-cgi: [Suricata] Building new sid-msg.map file for BLAZENET_ISP_1...
Nov 26 12:30:23 pfSense SuricataStartup32425: Suricata STOP for BLAZENET_ISP_1_WAN(27120_em0)...
Nov 26 12:30:25 pfSense kernel: em0: link state changed to DOWN
Nov 26 12:30:25 pfSense check_reload_status: Linkup starting em0
Nov 26 12:30:26 pfSense php-fpm: /rc.linkup: Hotplug event detected for BLAZENET_ISP_1(wan) static IP (X.X.X.X )
Nov 26 12:30:26 pfSense check_reload_status: Reloading filter
Nov 26 12:30:26 pfSense php-cgi: [Suricata] Suricata has restarted with your new set of rules...
Nov 26 12:30:26 pfSense php-cgi: [Suricata] The Rules update has finished.
Nov 26 12:30:26 pfSense SuricataStartup36834: Suricata START for BLAZENET_ISP_1_WAN(27120_em0)...
Nov 26 12:30:26 pfSense check_reload_status: Syncing firewall
Nov 26 12:30:44 pfSense check_reload_status: Linkup starting em0
Nov 26 12:30:44 pfSense kernel: em0: link state changed to UP
Nov 26 12:30:45 pfSense php-fpm: /rc.linkup: Hotplug event detected for BLAZENET_ISP_1(wan) static IP (X.X.X.X )
Nov 26 12:30:45 pfSense check_reload_status: rc.newwanip starting em0
Nov 26 12:30:45 pfSense check_reload_status: Reloading filter
Nov 26 12:30:46 pfSense php-fpm: /rc.newwanip: rc.newwanip: Info: starting on em0.
Nov 26 12:30:46 pfSense php-fpm: /rc.newwanip: rc.newwanip: on (IP address: X.X.X.X) (interface: BLAZENET_ISP_1[wan]) (real interface: em0).
Nov 26 12:30:46 pfSense check_reload_status: Reloading filter
Files
Updated by Srijan Nandi over 4 years ago
Suricata is running in INLINE IPS mode. Every time, when suricata is stopped or started, it does a link up/down. Is this normal behaviour with INLINE IPS mode? Can it be modified to not restart interfaces while suricata restarts.
Updated by Jim Pingle over 4 years ago
- Project changed from pfSense to pfSense Packages
- Category changed from Package System to Suricata
Updated by Danilo Zrenjanin about 4 years ago
You can set Suricata to "Live Reload" the new rules without restarting itself.
Enable this global option to prevent a hard restart of Suricata when the rules reload:
Suricate > Global Settings, Live Rule Swap on Update.
Updated by Bill Meeks about 4 years ago
Suricata running with Inline IPS Mode uses the netmap kernel device. When Suricata stops and restarts, that also stops and restarts the netmap kernel device resulting in the network interface cycling. This is not a bug. It's just the way netmap works.
You have two options: (1) abandon Inline IPS Mode and use Legacy Mode with libpcap; or (2) do what user Danilo Zrenjanin Zrenjanin suggested and enable the "Live Reload" option for Suricata rules updates on the GLOBAL SETTINGS tab. This will not physically stop and restart Suricata on rules updates but will instead signal the running daemon of the need to reload the rules.
Note that choosing option #2 above may somewhat increase temporary use of RAM as for a period of time Suricata needs to keep two complete copies of the enabled rules in memory.
Updated by Viktor Gurov about 4 years ago
- File suricata.yaml suricata.yaml added
- Status changed from Not a Bug to New
same issue on XG-1537 (pfSense 2.4.4-p3, suricata 4.1.6_3) with ix interface,
I found that killing suricata process (service stop) changes interface status to down ('no carrier' on the interface status page):
carp: 1@ix1: MASTER -> INIT (hardware interface down) kernel: carp: demoted by 240 to 240 (interface down) kernel: ix1: link state changed to DOWN
by "pkill -TERM -F /var/run/suricata_ix11234.pid" or "pkill -TERM -fn "suricata -i ix1"
but not in 100% of cases - sometime only 3rd or 4th restart 'down' the interface
also reproduced on APU.3B4 (2.4.5-RC, amd64) - but the interface did not remain in a “no carrier” state
SuricataStartup 45332 Suricata START for LAN(34549_igb0)... Squid_Alarm 46079 Squid is disabled, exiting. radiusd 35803 Ready to process requests php-fpm 18547 /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules. check_reload_status Linkup starting igb0 kernel igb0: link state changed to DOWN kernel igb0.33: link state changed to DOWN kernel igb0.99: link state changed to DOWN kernel igb0.111: link state changed to DOWN
<ips_mode>ips_mode_legacy</ips_mode> - in all tests
No such issue on VM or SG-1100 - but interfaces in both cases never go down
Updated by Bill Meeks about 4 years ago
If Suricata is running using Legacy Mode Blocking, then the libpcap library is used and bonded to the interface where Suricata is enabled. The custom blocking module is not involved in this process at all. The standard pcap-binding process is used within the Suricata binary. Perhaps something has changed in that code from upstream, or something has changed in the libpcap library ???
I can look through the commits for Suricata upstream to see if anything has been changed lately in the PCAP portion of the code.
Updated by Bill Meeks about 4 years ago
A look through the Suricata source code shows that the Suricata binary, when running in PCAP mode, will send explicit commands to the OS to disable checksum offloading on the physical interface. Perhaps that is now causing the OS kernel to cycle the physical interface ???
It appears the Suricata binary will first get the current operating flags for the physical interface, and if checksum offloading is enabled it will be forcibly disabled. Later, when Suricata stops, the binary will restore the previous operating flags for the physical interface.
So how about repeating the above tests but with the checksum offloading and TCP segmentation offloading disabled within pfSense? That way Suricata will not attempt to change the offloading as it will be already correctly set.
Updated by Marcos M over 1 year ago
- Status changed from New to Closed
Interfaces are now reloaded live without bringing down the interface.