pfSense » pfSense Packages

To quote valete3. . .

_"Emerging threats released out of band rules update to resolve.

https://community.emergingthreats.net/t/ruleset-update-summary-2023-06-15-v10349/656/4

This problem was due to two rules using the syntax flow:stateless,to_server for the snort version of two of the SEASPY rules. Snort does not like having flow:stateless combined with other flow options and throws an error. The error isn’t formatted like any of the other errors Snort typically throws regarding rule syntax errors, and our QA systems missed it. Our QA system has been updated to account for this error, and we’ve released an emergency out of band update that is available now to fix this problem.

We apologize for any inconvenience this has caused you or any other netgate customers, and have made necessary precautions to prevent it from happening in the future. If there is anything else I can do for you, please let me know."_

Leading to such a update condition can and does offline all of the IPS/IDS security

To quote bemeeks,

" _This will have to be fixed by the Emerging Threats rule writers. They will release an updated rules archive for those rules pretty soon is my guess. As others have mentioned, you can temporarily work around it by disabling the culprit rules.

If something similar happens in the future, the error message can help you narrow down the rule. For this case, the error message gives you the rules file being processed and what line number in the file caused a problem.

/usr/local/etc/snort/snort_3612_ix1/rules/snort.rules:26779

Open the given file in a text editor and locate line #26779. The rule on that line will be the one causing the problem.

The Snort package collects all the active enabled rules and writes them into a snort.rules file for each configured Snort interface. Each configured interface has its configuration info provided in a separate subdirectory under /usr/local/etc/snort/._ "

Idea for fix:

What about it keeps the old ruleset if it fails with such a condition reload the old rules and stop for a period of a sometime like 8 hours and try again. Log the errors so everyone can see with a log that says loading prior rule set with a date and time? It pointed to a rule after it updated, so it should in theory be able to catch the error and do an if statement.

Main issue: Snort fails completely open within this situation. Snort does not function at all during this.

This is not a bug. The problem described here was caused by a faulty rules update file produced and distributed by a third-party (in this instance, Emerging Threats). The Emerging Threats team indentified the issue and released a hotfix update within 12 hours or so.

This is perhaps a feature request, but it's not a bug in my opinion. There is nothing wrong with the functionality of the Snort package. As soon as the rule vendor released an updated archive and it was installed on the firewall, Snort started normally.

Thanks for the reply Bill Meeks,

Please let me attempt to pitch this one more time as a bug and not a feature to you.

I agree Emerging Threats rule sets caused this issue. Again, Snort failed wide open because of it.

Leading to, the timing of this with relationships to what CISA reported today is of major concern. I can't imagine a worse time to have this fail completely open. Many vendors put there trust into Netgate and pfSense. Let's agree that it should never fail open because of a rule set update from a third party like Emerging Threats. The Snort package itself has a GUI radio button built in that allows users to enable and start using Emerging threats rule sets. You stated it took 12 hours for the hotfix, I assume some small office users had Snort failed open for that full time frame.

Have a good weekend. I have attached some references with the attack that occurred at the same time this failed open. The nation state actors saw a window and went for it.

Ref:
https://www.reuters.com/world/us/us-government-agencies-hit-global-cyber-attack-cnn-2023-06-15/
https://www.cisa.gov/news-events/alerts/2023/06/15/progress-software-releases-security-advisory-moveit-transfer-vulnerability

I'll chime in with another view point that I find disturbing. Not classifying this as a bug, or at the least a security enhancement request, keeps Snort/pfSense open to a known exploit. What if a bad actor manages to get a rule published in ET or Snort rulesets, either because he was on the team or outside it through their communities, that is known to instantly fatally crash Snort on everyone's pfSense instance. With the intention of targeting specific known vulnerable networks right afterwards. We saw it took ET a day to respond to this, even though I reported it to their support within the hour of them publishing it, I don't know how many peoples systems were vulnerable during that time (especially ones who didn't monitor or have alerts setup). However I imagine it was a lot - and this was kind of an idea-best-case scenario IMO. Yes, we all know the obvious high availability and QA environment argument of why this shouldn't bring a production firewall down, but I think we should also realize most instances of pfSense probably are not setup like that and it's a rather unfeasible of a suggestion for a lot.

I think leaving this avenue of attack open is silly, unless there's no way to prevent it pfSense/Snort side. I obviously do not know the codebase but I can infer that pfSense/Snort was able to return the line # of the rule that it stopped on when loading. If it can identify the failed line, it can easily pull the SID from that line. Then I imagine if it knows the SID it can auto-disable the rule (or remove that line from the file - not sure but I think that file is dynamically built in runtime). I don't really like the idea of the system auto disabling/enabling rules (at least without alerting), but I suppose I like the idea of Snort being COMPLETELY disabled much much less, and rule enabling/disabling is also controlled somewhat by the rulesets I believe so it may be par-the-course. I don't really know if this is something that could be handled by Snort, or pfSense, since pfSense seems to control the service and rulesets it may be the one who should handle a failed rule, and auto-disabling a failed rule could even be a configuration option in pfSense. Would pushing the issue to the pfSense team be more viable?

Another suggestion I would have is perhaps returning a better error in the GUI/logs, so it's easier to discover what the cause of fatal errors like this were. To find this you had to pull the error message from the GUI log and get the line number (after realizing it wasn't a confusing reference to a SID number that's formatted similarly). Then enable ssh and ssh into the firewall, goto shell, then awk the file or whatever method to retrieve the specific line in question. Then you can get the SID yourself, and hope it says something identifying in the message portion that tells you what category that rule falls under, if not you might have to google the SID to figure out which category it is in. Then you have to go back to pfsense GUI -> Snort -> Interface -> Wan Rules -> Category -> disable the SID# you pulled out above. It was a bit annoying to work through that, and I imagine a lot of people didn't. Returning the SID + Category, or full rule, would've at least greatly helped the average user manually quickly identify and disable the offending rule.

The Snort package on pfSense is an open source volunteer maintained contribution. The source code for both the GUI and binary portions of the package are freely available on Github at https://github.com/pfsense/FreeBSD-ports/tree/devel/security/pfSense-pkg-snort and https://github.com/pfsense/FreeBSD/ports/tree/devel/security/snort. Contributions via pull requests are welcomed.

As the package maintainer, I do not see this as a bug. It is a near impossible task to anticipate and write PHP GUI code to respond to every conceivable fatal error the Snort binary can produce. There can be a multitude of problems from memory issues, preprocessor issues to rule syntax that prevent Snort from starting successfully. Unlike the Suricata binary, the Snort binary will fatal error exit on any of a multitude of errors encountered during startup including rule syntax issues. Suricata, on the other hand, will log error messages, skip the offending rule or rules, and move on to start succesfully.

The solution to the reliability issue is simple -- turn off automatic unattended rule updates. Instead, update rules manually when you are present to monitor the process and take appropriate action. Even better, have a test bed where you download and test updates, then after verifying things are okay, you manually update your production system. No mission critical system should ever be set to automatically update itself unattended as any update could introduce a system failure.

If anything, this problem would be best addressed by submitting it upstream to the Snort binary team. They control the logic of the binary. The IDS/IPS packages are really controlled and contained within the respective binaries which are maintained by upstream third parties. The GUI is simply a PHP wrapper piece that creates the text configuration file the binary uses at startup and performs some rudimentary maintenance functions (such as downloading new rules).

In terms of logging, the GUI can only log what the binary outputs as errors. Changing the Snort error message in this case would require recoding the binary.

https://github.com/pfsense/FreeBSD-ports/tree/devel/security/snort

Thanks for the reply again,

I wanted to ask, with Java at Sierra college while I was working on my AA we were taught how to write code with "try and catch" for conditional error catching for specific needs. I am sure you know this was to keep the system from crashing. Is there no way to do "try catch" for the rules when they are loading with C+? My primary concern was Snort failed Open and not Closed on functional system during a ruleset update. Keep in mind I am still working on my degree in computer science currently. But I do have my AA in cyber security. I just thought it was the same process within error catching here. Leading to, when you said it would require a full binary change. I think I understand this as PfSense only adds the GUI for the Snort package? Correct me if I am
wrong? You said Suricata has this hypothetical "try" the rule "catch" bad rules/skip to next rule code, and Snort does not?

side note: I think found out why my codespaces environment won't run, I have the free account. It is similar to https://replit.com/ as the ide is web based. Codespaces would never start and just say error when creating a workspace.