Project

General

Profile

Actions

Bug #6235

closed

Snort sometimes crashes during rule update process (specifically related to VRT .so rule update?)

Added by Andrew W over 5 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Category:
Snort
Target version:
-
Start date:
04/22/2016
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Affected Version:
Affected Plus Version:
Affected Architecture:

Description

Snort occasionally crashes during the rule update process and doesn't start again until I manually restart it via the GUI. When the crash occurs, it usually crashes with signal 11 (SIGSEGV), but most recently it crashed with signal 4 (SIGILL). The logs from the latest crash are as follows:

Apr 20 04:00:01 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Snort VRT rules posted. Downloading snortrules-snapshot-2980.tar.gz...
Apr 20 04:00:13 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort VRT rules file update downloaded successfully
Apr 20 04:00:13 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort OpenAppID detectors are up to date...
Apr 20 04:00:14 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Apr 20 04:00:15 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Apr 20 04:00:26 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: LAN ...
Apr 20 04:00:28 x kernel: pid 65324 (snort), uid 0: exited on signal 4
Apr 20 04:00:28 x kernel: igb1: promiscuous mode disabled
Apr 20 04:00:42 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: LAN...
Apr 20 04:00:44 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for LAN...
Apr 20 04:00:51 x php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
Apr 20 04:00:51 x check_reload_status: Syncing firewall
[End of logs for Apr 20]

From /var/log/snort/snort_rules_update.log:

Starting rules update...  Time: 2016-04-20 04:00:00
        Downloading Snort VRT rules md5 file snortrules-snapshot-2980.tar.gz.md5...
        Checking Snort VRT rules md5 file...
        There is a new set of Snort VRT rules posted.
        Downloading file 'snortrules-snapshot-2980.tar.gz'...
        Done downloading rules file.
        Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
        Checking Snort OpenAppID detectors md5 file...
        Snort OpenAppID detectors are up to date.
        Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
        Checking Emerging Threats Open rules md5 file...
        There is a new set of Emerging Threats Open rules posted.
        Downloading file 'emerging.rules.tar.gz'...
        Done downloading rules file.
        Extracting and installing Snort VRT rules...
        Using Snort VRT precompiled SO rules for FreeBSD-10-0 ...
        Installation of Snort VRT rules completed.
        Extracting and installing Emerging Threats Open rules...
        Installation of Emerging Threats Open rules completed.
        Copying new config and map files...
        Updating rules configuration for: LAN ...
The Rules update has finished.  Time: 2016-04-20 04:00:51

Notice above that the Snort VRT precompiled SO rules were updated - on previous days that these rules were not updated, the snort update process completed successfully without the snort process crashing. I wonder if the .so's are being modified such that the running snort process will crash upon trying to call a function provided by the updated libraries (after the function address has been resolved based on the old .so).

PfSense Machine Specs:

Netgate RCC-DFF    
Intel(R) Atom(TM) CPU C2338 @ 1.74GHz
2.3-RELEASE (amd64) 
built on Mon Apr 11 18:28:29 CDT 2016 
FreeBSD 10.3-RELEASE

I've had these problems for the last few months (running 2.2.6 as well.)

Please let me know what additional information I can provide. Thanks!


Files

snort_check_for_rule_updates.patch (2.18 KB) snort_check_for_rule_updates.patch Sander Peterse, 04/16/2021 03:04 AM
Actions #1

Updated by Chris Buechler over 5 years ago

  • Project changed from pfSense to pfSense Packages
  • Category set to Snort
Actions #2

Updated by Bill Meeks over 5 years ago

The rules update process will restart any running Snort processes at the end of the update. However, I guess it is possible that the currently running Snort process might try to access the shared object (*.so) rules during the instant of update and have a problem with that. Purely conjecture, though, on my part.

I have not noticed this problem on my personal system, but it is a home firewall and does not see much traffic.

This one might be hard to nail down since it is intermittent. Keep an eye on it to see if it happens again and if the crash corresponds with a fresh set of VRT rules. Those shared object rules will get updated on each Snort VRT rules package update. Those generally come out on Tuesdays and Thursdays US Eastern Time.

Bill

Actions #3

Updated by Andrew W over 5 years ago

This happens fairly frequently for me - it crashed again on Apr. 22nd when pulling in the latest VRT rules (SIGSEGV).

If I can make snort generate a core dump file the next time it crashes, that might point to whether it is crashing when attempting to call a function in one of the .so's. I haven't figured out exactly how to make snort generate a coredump when it crashes, though. From reading online, hopefully the following will work?

sysctl kern.corefile=/tmp/%N.core
sysctl kern.capmode_coredump=1
sysctl kern.sugid_coredump=1
Actions #4

Updated by Frank Seidel over 3 years ago

I also have the same problem.. nearly every night on the rule update process the snort service
dies or isn't coming up again after the updates ran.
(running pfSense community edition 2.4.3-RELEASE-p1 (amd64) with snort 3.2.9.6_1)

Jul 20 02:06:50 check_reload_status Syncing firewall
Jul 20 02:06:49 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
Jul 20 02:06:42 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for LAN...
Jul 20 02:06:39 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: LAN...
Jul 20 02:06:15 kernel igb2: promiscuous mode disabled
Jul 20 02:06:15 kernel pid 2526 (snort), uid 0: exited on signal 4
Jul 20 02:06:09 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: LAN ...
Jul 20 02:05:33 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Jul 20 02:05:29 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Jul 20 02:05:28 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort OpenAppID RULES detectors file update downloaded successfully
Jul 20 02:05:27 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Snort OpenAppID RULES detectors posted. Downloading appid_rules.tar.gz...
Jul 20 02:05:27 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort OpenAppID detectors are up to date...
Jul 20 02:05:25 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort Subscriber rules file update downloaded successfully
Jul 20 02:05:05 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Snort Subscriber rules posted. Downloading snortrules-snapshot-29111.tar.gz...

Actions #5

Updated by Sander Peterse 8 months ago

This issue still is still there. It happened last night to 2 of our PFSense boxes. Snort crashes due to the update process and is not restarted afterwards as the update process assumes it should not be running.

But I think can be fixed quite easily. I have attached a patch file.

What this patch changes: It checks (and stores) the Snort running start before doing any changes. It uses the stored running state afterwards to determine if it should restart Snort.

Actions #6

Updated by Bill Meeks 8 months ago

Thank you for the suggested patch, but I think the rules update logic is going to need additional changes due to the inclusion of the Inline IPS option. With Inline IPS operation, any Snort stop/start operation will also result in the physical interface cycling down then back up in pfSense. This can interrupt other processes, so some Snort users want the option of Live Rules updates which, instead of stopping and restarting Snort, will send the process a signal to "live reload" the new rules.

The SO rules are special in that they are precompiled binary libraries (you probably already know this). And they are distributed pre-compiled for specific OS versions. There is a FreeBSD-12 set of SO rules, but I suspect they are compiled for RELEASE while pfSense is based on STABLE. Also not sure which 12.x version they are compiled for. So there is a bit of a potential wildcard there that might be a cause of random segfaults.

I will make the rules update process a bit more robust by incorporating your idea of saving the "entry state" for service running or not running. But I will also add logic to make sure the process accomodates Inline IPS folks who want to use the "live reload" option. Look for these changes in the near future.

Actions #7

Updated by Bill Meeks 5 months ago

The Snort GUI package now has additional logic to ensure running Snort interfaces at the start of a rules update cycle are started/restarted when the rules update cycle is completed. See this Pull Request: https://github.com/pfsense/FreeBSD-ports/pull/1075.

Once this Pull Request is merged, this issue can be closed.

Actions #8

Updated by Renato Botelho 5 months ago

  • Status changed from New to Resolved
  • Assignee set to Renato Botelho

PR has been merged

Actions

Also available in: Atom PDF