Bug #5848
closed
Downloaded rules data validation
0%
Description
Occasionally, data is not properly downloaded from internet based sources, and rules cannot be generated with errors similar to:
php-fpm[]: /rc.filter_configure_sync: New alert found: There were error(s) loading the rules: /tmp/rules.debug:19: file "/etc/bogons" contains bad data - The line in question reads [19]: table <bogons> persist file "/etc/bogons" php-fpm[]: /rc.filter_configure_sync: New alert found: There were error(s) loading the rules: /tmp/rules.debug:202: syntax error - The line in question reads [X]: pass in quick on $IPsec inet proto tcp from $Network1 to $Network2 port $Ports tracker 123 flags S/SA keep state label "USER_RULE"
- If there are errors downloading the data, configurable attempts should be made to retry until successful data downloads or the number of attempts is exhausted. (with a configurable pause between attempts)
- If data is not in the expected format, for ports or cidr blocks, it should not be saved. (i.e. a 404/503 error text)
- If the rule points to an empty (or invalid) data file, it should not be loaded.
Updated by Jim Thompson almost 9 years ago
- Status changed from New to Feedback
- Assignee set to Chris Buechler
What "Internet sources"?
Updated by Alex Vergilis almost 9 years ago
RE "internet sources"?
From the 2 provided log samples of failures that occur across all of my tracked systems:
1 - Bogon Networks
Interfaces > Interface > Block bogon networks
Loaded by cron, for example: /etc/rc.update_bogons.sh
2 - Aliases
Firewall > Aliases > URL (IPs)
Firewall > Aliases > URL (Ports)
Firewall > Aliases > URL Table (IPs)
Firewall > Aliases > URL Table (Ports)
Loaded by cron, for example: /etc/rc.update_urltables
Updated by Chris Buechler almost 9 years ago
- Status changed from Feedback to Confirmed
- Affected Version changed from 2.2.5 to All
Updated by Chris Buechler almost 9 years ago
- Status changed from Confirmed to Feedback
I've been through a slew of circumstances that were broken before and are all fine now. There are multiple levels of additional verification that prevent invalid data from getting into a table or the ruleset.
The only part I'm not sure about is how you ended up with invalid data in /etc/bogons, do you have the contents of that file from the time? That's not loosely validated like the URL Table aliases were. It has to fetch the list and its md5 via HTTPS, with certificate verification, and if the md5 doesn't match it doesn't put the file into place. Server-side validation also ensures the file's contents are good.
Updated by Alex Vergilis almost 9 years ago
Unfortunately, I do not have a sample I can provide. The file contents have been overwritten with other valid downloaded data after the failures.
Is there an md5 validity check for URL Table aliases that can be implemented to minimize errors?
Updated by Chris Buechler almost 9 years ago
- Status changed from Feedback to Resolved
This all works.
Alex Vergilis wrote:
Is there an md5 validity check for URL Table aliases that can be implemented to minimize errors?
not at this time. With all the validation here now I don't think it would help, shouldn't be any circumstance where a web server delivers a file with a 200 return code that wouldn't match a md5.