pfSense » pfSense Packages

I receive the errors on the following versions of pfSense:

2.4.4-RELEASE-p1 (amd64) (Netgate hardware) with Suricata 4.0.13_11 in IDS mode
2.4.4-RELEASE-p1 (amd64) ( " " ) after re-installing Suricata 4.0.13_11 (keeping the configuration) in IDS mode

and

2.4.5.a.20181211.1622 (not Netgate hardware) with Suricata 4.0.13_11 in IPS mode (as well as a prior version, not recorded but if memory serves me 2 weeks old)
2.4.5.a.20181212.1603 ( " " ) after updating to latest pfSense, with Suricata 4.0.13_11 in IPS mode

I have not been able to reproduce this error in any of my testing. I have tested updated an existing Suricata installation and also tested with a complete green field fresh install. It works every time with the ET Open rules without errors.

Your problem is caused by the master reference.config getting removed. I have no idea how that happened to you, but that is the cause of this error. To fix it, remove the Suricata package and install it again. Your previous settings will be preserved.

This issue should be closed.

I have uninstalled Suricata without preserving settings and re-installed from scratch. I still see these errors. I don't see as many now, but I have not enabled as many of the categories in Suricata -> Interface -> Categories -> ET Open Rules.

On the same page, I use: Snort IPS Policy selection -> Use IPS Policy (checked), Use rules from one of three pre-defined Snort IPS policies (different ones for different interfaces).

I have clicked on Diagnostics -> Backup & Restore -> Backup & Restore -> Package Functions -> Reinstall Packages.

I now see the very large number of errors (due to unknown reference keys "bid" and "md5").

The re-install log covering Suricata is:

The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
pfSense-pkg-suricata-4.0.13_11 [pfSense]

Number of packages to be reinstalled: 1

134 KiB to be downloaded.
[1/1] Fetching pfSense-pkg-suricata-4.0.13_11.txz: .......... done
Checking integrity... done (0 conflicting)
[1/1] Reinstalling pfSense-pkg-suricata-4.0.13_11...
[1/1] Extracting pfSense-pkg-suricata-4.0.13_11: .......... done
Removing suricata components...
Menu items... done.
Services... done.
Loading package instructions...
Saving updated package information...
overwrite!
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...Saved settings detected...
Migrating settings to new configuration... done.
Downloading Emerging Threats Open rules md5 file... done.
There is a new set of Emerging Threats Open rules posted. Downloading... done.
Downloading Snort VRT rules md5 file... done.
There is a new set of Snort rules posted. Downloading... done.
Installing Emerging Threats Open rules... done.
Installing Snort rules... done.
Updating rules configuration for: WAN ... done.
Updating rules configuration for: LAN ... done.
Updating rules configuration for: L2 ... done.
Updating rules configuration for: L10 ... done.
Updating rules configuration for: L100 ... done.
Updating rules configuration for: L110 ... done.
Updating rules configuration for: L120 ... done.
Updating rules configuration for: L200 ... done.
Updating rules configuration for: L210 ... done.
Updating rules configuration for: L220 ... done.
Cleaning up after rules extraction... done.
The Rules update has finished.
Generating suricata.yaml configuration file from saved settings.
Generating YAML configuration file for WAN... done.
Generating YAML configuration file for LAN... done.
Generating YAML configuration file for L2... done.
Generating YAML configuration file for L10... done.
Generating YAML configuration file for L100... done.
Generating YAML configuration file for L110... done.
Generating YAML configuration file for L120... done.
Generating YAML configuration file for L200... done.
Generating YAML configuration file for L210... done.
Generating YAML configuration file for L220... done.
Finished rebuilding Suricata configuration from saved settings.
Setting package version in configuration file.
done.
Executing custom_php_resync_config_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.

Cleaning up cache... done.

Success

cat /usr/local/etc/suricata/reference.config

config reference: arachNIDS http://www.whitehats.com/info/IDS
config reference: bugtraq http://www.securityfocus.com/bid/
config reference: cve http://cve.mitre.org/cgi-bin/cvename.cgi?name=
config reference: McAfee http://vil.nai.com/vil/content/v_
config reference: msb http://technet.microsoft.com/en-us/security/bulletin/
config reference: nessus http://cgi.nessus.org/plugins/dump.php3?id=
config reference: osvdb http://osvdb.org/show/osvdb/
config reference: url http://

config reference: McAfee http://vil.nai.com/vil/content/v_

I am unable to resolve "vil.nai.com". Problem?

I have tried System -> Package Manager -> Installed Packages -> Suricata -> clicked Reinstall. Same issue.

Will try resetting to Factory Default and restoring a backup.

Note: I have not closed the bug report as I continue to have the issue.

Bug report #9202 was closed as a duplicate. However, bug #9202 relates to the failure of re-install options from fixing this bug #9195. Clearly two different bugs.

Apparently, Bill Meeks has dismissed this bug because he is unable to reproduce it. Further, Jim Pringle has dismissed the new bug related to how this bug can't be fixed with Bill Meeks' pro-offered repair option.

Thanks pfSense team for your attention in dismissing this bug.

This issue is still open. If a proposed workaround for this issue didn't fix it doesn't make that a new issue. It's still this same problem, whatever that problem may be.

If this issue was "dismissed", it would be closed/rejected/etc. It isn't. Keeping all the notes for the same issue together on a single open issue is best.

My bug report that re-installing Suricata does not restore important configuration files to their default settings was dismissed as a duplicate of this bug. I believe it is a separate issue from this issue that the errors exist. One is the errors, the other is that the responsible file is not restored on re-install.

I have performed "Diagnostics -> Factory Defaults" then clicked on Factory Reset. Before the Factory Reset, I removed Suricata with "Keep Suricata Settings After Deinstall" not checked.

After the Factory Reset, there is now a more complete version of "reference.config" which there was not with all of the other re-install options (dismissed bug #9120):

cat /usr/local/etc/suricata/reference.config
config reference: arachNIDS http://www.whitehats.com/info/IDS
config reference: bid http://www.securityfocus.com/bid/
config reference: bugtraq http://www.securityfocus.com/bid/
config reference: cve http://cve.mitre.org/cgi-bin/cvename.cgi?name=
config reference: et http://doc.emergingthreats.net/
config reference: etpro http://doc.emergingthreatspro.com/
config reference: exploitdb http://www.exploit-db.com/exploits/
config reference: McAfee http://vil.nai.com/vil/content/v_
config reference: md5 http://www.threatexpert.com/report.aspx?md5=
config reference: msb http://technet.microsoft.com/en-us/security/bulletin/
config reference: msft http://technet.microsoft.com/security/bulletin/
config reference: nessus http://cgi.nessus.org/plugins/dump.php3?id=
config reference: openpacket https://www.openpacket.org/capture/grab/
config reference: osvdb http://osvdb.org/show/osvdb/
config reference: secunia http://www.secunia.com/advisories/
config reference: securitytracker http://securitytracker.com/id?
config reference: telus http://
config reference: threatexpert http://www.threatexpert.com/report.aspx?md5=
config reference: url http://
config reference: xforce http://xforce.iss.net/xforce/xfdb/

There were none of the errors until I checked "Suricata IDS -> Interface xxx -> xxx Categories -> Snort IPS Policy selection -> Use IPS Policy". It does not appear relevant whether the "IPS Policy Mode" is "Alert" or "Drop".

Connectivity policy produces one error:

<Error> -- [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(102)] - unknown rule keyword 'byte_math'.
16/12/2018 -- 17:26:56 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "drop tcp $EXTERNAL_NET any -> $HOME_NET 4786 (msg:"SERVER-OTHER Cisco Smart Install invalid init discovery message denial of service attempt"; flow:to_server,established; content:"|00 00 00|"; depth:3; content:"|00 00 00 07|"; within:4; distance:5; fast_pattern; content:"|00 00 00 01|"; within:4; distance:4; byte_math:bytes 4,offset 0,oper +,rvalue 8,result sub_len_plus_eight,relative; byte_test:4,!=,sub_len_plus_eight,-8,relative; metadata:policy balanced-ips drop, policy connectivity-ips drop, policy max-detect-ips drop, policy security-ips drop; reference:cve,2018-0171; reference:url,tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180328-smi2; classtype:attempted-dos; sid:46468; rev:1;)" from file /usr/local/etc/suricata/suricata_42039_em2/rules/suricata.rules at line 10819

Balanced produces 53 errors. Security produces 83 errors. Maximum Detection produces 78 errors (fewer than Security).

I currently have "Snort free Registered User or paid Subscriber rules" checked (it is the free registered version). I have tried Snort Rules:

snortrules-snapshot-2983.tar.gz (61 rules failed)
snortrules-snapshot-2990.tar.gz
snortrules-snapshot-29110.tar.gz
snortrules-snapshot-29111.tar.gz
snortrules-snapshot-29120.tar.gz
snortrules-snapshot-3000.tar.gz (14828 rules failed; clearly not v3 ready)

(Peculiar: after using "snortrules-snapshot-2983.tar.gz", which I tried last, the number of errors for Maximum Security with "snortrules-snapshot-29111.tar.gz" dropped to 61 errors, the same number with "2983". With the exception of "snortrules-snapshot-3000.tar.gz", the rules all had 78 errors until "2983" was installed.)

The errors appear therefore to be due to selecting a Snort IPS Policy. I cannot confirm if my "reference.config" file is correct, but after a Factory Reset, there were additional entries. These entries were not restored when re-installing Suricata (bug #9120).

The errors now appear to be due to illegal rules instead of "unknown reference key", with the exception an unknown rule keyword:

[100131] <Error> -- [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(102)] - unknown rule keyword 'http_raw_cookie'.
[100131] <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "drop tcp $EXTERNAL_NET $HTTP_PORTS > $HOME_NET any (msg:"MALWARE-OTHER Win.Trojan.Zeus Spam 2013 dated zip/exe HTTP Response - potential malware download"; flow:to_client,established; content:"-2013.zip|0D 0A|"; fast_pattern:only; content:"-2013.zip|0D 0A|"; http_header; content:""; within:1; distance:-14; http_header; file_data; content:"-2013.exe"; content:"-"; within:1; distance:-14; metadata:impact_flag red, policy balanced-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/2eff3ee6ac7f5bf85e4ebcbe51974d0708cef666581ef1385c628233614b22c0/analysis/; classtype:trojan-activity; sid:26470; rev:1;)" from file /usr/local/etc/suricata/suricata_42039_em2/rules/suricata.rules at line 18673

I presume the original issue is resolved (with a Factory Reset albeit not with a re-install of Suricata).

I have made at least a couple of posts on the pfSense IDS/IPS forum about this: snortrules-snapshot-3000.tar.gz (14828 rules failed; clearly not v3 ready). You cannot use Snort3 rules with Suricata. Also remember that Suricata is not Snort. Snort rules are created and optimized for Snort. Suricata can understand most of the Snort rules, but not all; hence the rule loading errors. How many errors depends on which version of Snort rules you try and which rules out of that version are enabled. The Emerging Threats folks support Suricata and have a tree of their rules that is optimized for Suricata.

I ran into this issue as well after having tried the Snort3 rules and reverted to 2.9 - Suricata is far pickier about keyword order than Snort is.

In my case reference.config was simply missing as were all of the default Suricata rules were missing from /usr/local/share/suricata/rules. I force reinstalled the suricata package which resolved the missing rules, but no amount of regenerating the config would restore reference.config. I ended up manually restoring reference.config by copying /usr/local/etc/suricata/reference.config.sample and this resolved most of the issues I was seeing.

Does reference.config get regenerated every time the configuration is built? My experience would suggest that it does not although it seems as though it should.

Snort3 rules are incompatible with Suricata 4.x. If you install those rules, they will overwrite some critical configuration files (reference.config being one of them) and bork your Suricata installation. Also, since the modified reference.config file is marked as "user modified" because it was not changed by the pkg utility, when you uninstall Suricata that modified file is not removed. So it is there waiting to screw up Suricata when it is installed again. Your solution of manually copying over the sample reference.config file is the only one.

Bill

Possibly a separate issue, but I am unable to resolve warnings that "app-layer-events.rules seems to be missing!!! Please verify rules files have been downloaded, then go to the Categories tab and save the rule set again." There are similar warnings for "decoder-events.rules", "dnp3-events.rules", and "dns-events.rules". I have tried reinstalling Suricata and force reinstalling all packages. I have tried unchecking, "Snort IPS Policy selection -> Use IPS Policy -> Use rules from one of three pre-defined Snort IPS policies".

I resolved the issue discussed by John Silva with his suggestion (confirmed by Bill Meeks). I don't see similar copies of sample of these rules.

Uninstall without keeping settings and re-install restored the missing rules. Much work ahead to restore to its previous glory.

P Law wrote:

Possibly a separate issue, but I am unable to resolve warnings that "app-layer-events.rules seems to be missing!!! Please verify rules files have been downloaded, then go to the Categories tab and save the rule set again." There are similar warnings for "decoder-events.rules", "dnp3-events.rules", and "dns-events.rules". I have tried reinstalling Suricata and force reinstalling all packages. I have tried unchecking, "Snort IPS Policy selection -> Use IPS Policy -> Use rules from one of three pre-defined Snort IPS policies".

I resolved the issue discussed by John Silva with his suggestion (confirmed by Bill Meeks). I don't see similar copies of sample of these rules.

Those rules are part of the base binary package installation. There was a bug many, many versions back where the build missed including some newer built-in events rules files. "app-layer-events.rules" was one of them. Sounds like you had a "confused" configuration section in the firewall's config.xml file where the Suricata configuration is stored.

If you still have a config.xml backup from your previous configuration, you can restore things like your rules and other settings by copying over the relevant parts of the old config.xml into the new config.xml. You just have to be really careful and not overwrite formatting information and element tags. If you want to try this, make sure you keep a backup of your "new" current configuration before editing it! If you mess up the config.xml file, the firewall can potentially fail to boot up.

A fix for this issue has been incorporated into the Suricata GUI package in version 4.1.4_2. The pull request is posted here: https://github.com/pfsense/FreeBSD-ports/pull/647.

This issue can be closed and marked RESOLVED.