Downloaded rules data validation
Occasionally, data is not properly downloaded from internet based sources, and rules cannot be generated with errors similar to:
php-fpm: /rc.filter_configure_sync: New alert found: There were error(s) loading the rules: /tmp/rules.debug:19: file "/etc/bogons" contains bad data - The line in question reads : table <bogons> persist file "/etc/bogons" php-fpm: /rc.filter_configure_sync: New alert found: There were error(s) loading the rules: /tmp/rules.debug:202: syntax error - The line in question reads [X]: pass in quick on $IPsec inet proto tcp from $Network1 to $Network2 port $Ports tracker 123 flags S/SA keep state label "USER_RULE"
- If there are errors downloading the data, configurable attempts should be made to retry until successful data downloads or the number of attempts is exhausted. (with a configurable pause between attempts)
- If data is not in the expected format, for ports or cidr blocks, it should not be saved. (i.e. a 404/503 error text)
- If the rule points to an empty (or invalid) data file, it should not be loaded.
Return false in download_file rather than the failed status code when a download fails. Return would always evaluate to true previously though other parts of the code expect a false value when a download fails. related to Ticket #5848
Don't try to process aliases whose downloads have failed in update_alias_url_data(). Ticket #5848
Return false if download fails in process_alias_urltable so input validation based upon it works. Ticket #5848
Parse URL Table alias downloads with parse_aliases_file to ensure only valid contents. Ticket #5848
add an option to only validate URL tables, so stray files aren't left behind when input validation fails. Ticket #5848
#2 Updated by Alex Vergilis about 5 years ago
RE "internet sources"?
From the 2 provided log samples of failures that occur across all of my tracked systems:
1 - Bogon Networks
Interfaces > Interface > Block bogon networks
Loaded by cron, for example: /etc/rc.update_bogons.sh
2 - Aliases
Firewall > Aliases > URL (IPs)
Firewall > Aliases > URL (Ports)
Firewall > Aliases > URL Table (IPs)
Firewall > Aliases > URL Table (Ports)
Loaded by cron, for example: /etc/rc.update_urltables
#4 Updated by Chris Buechler about 5 years ago
- Status changed from Confirmed to Feedback
I've been through a slew of circumstances that were broken before and are all fine now. There are multiple levels of additional verification that prevent invalid data from getting into a table or the ruleset.
The only part I'm not sure about is how you ended up with invalid data in /etc/bogons, do you have the contents of that file from the time? That's not loosely validated like the URL Table aliases were. It has to fetch the list and its md5 via HTTPS, with certificate verification, and if the md5 doesn't match it doesn't put the file into place. Server-side validation also ensures the file's contents are good.
#6 Updated by Chris Buechler about 5 years ago
- Status changed from Feedback to Resolved
This all works.
Alex Vergilis wrote:
Is there an md5 validity check for URL Table aliases that can be implemented to minimize errors?
not at this time. With all the validation here now I don't think it would help, shouldn't be any circumstance where a web server delivers a file with a 200 return code that wouldn't match a md5.