Project

General

Profile

Actions

Bug #5848

closed

Downloaded rules data validation

Added by Alex Vergilis over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Category:
Rules / NAT
Target version:
Start date:
02/06/2016
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
All
Affected Architecture:

Description

Occasionally, data is not properly downloaded from internet based sources, and rules cannot be generated with errors similar to:

php-fpm[]: /rc.filter_configure_sync: New alert found: There were error(s) loading the rules: /tmp/rules.debug:19: file "/etc/bogons" contains bad data - The line in question reads [19]: table <bogons> persist file "/etc/bogons" 
php-fpm[]: /rc.filter_configure_sync: New alert found: There were error(s) loading the rules: /tmp/rules.debug:202: syntax error - The line in question reads [X]: pass  in  quick  on $IPsec inet proto tcp  from $Network1 to $Network2 port $Ports tracker 123 flags S/SA keep state  label "USER_RULE" 
  • If there are errors downloading the data, configurable attempts should be made to retry until successful data downloads or the number of attempts is exhausted. (with a configurable pause between attempts)
  • If data is not in the expected format, for ports or cidr blocks, it should not be saved. (i.e. a 404/503 error text)
  • If the rule points to an empty (or invalid) data file, it should not be loaded.
Actions #1

Updated by Jim Thompson over 8 years ago

  • Status changed from New to Feedback
  • Assignee set to Chris Buechler

What "Internet sources"?

Actions #2

Updated by Alex Vergilis over 8 years ago

RE "internet sources"?

From the 2 provided log samples of failures that occur across all of my tracked systems:

1 - Bogon Networks

Interfaces > Interface > Block bogon networks

Loaded by cron, for example: /etc/rc.update_bogons.sh

2 - Aliases

Firewall > Aliases > URL (IPs)
Firewall > Aliases > URL (Ports)
Firewall > Aliases > URL Table (IPs)
Firewall > Aliases > URL Table (Ports)

Loaded by cron, for example: /etc/rc.update_urltables

Actions #3

Updated by Chris Buechler over 8 years ago

  • Status changed from Feedback to Confirmed
  • Affected Version changed from 2.2.5 to All
Actions #4

Updated by Chris Buechler over 8 years ago

  • Status changed from Confirmed to Feedback

I've been through a slew of circumstances that were broken before and are all fine now. There are multiple levels of additional verification that prevent invalid data from getting into a table or the ruleset.

The only part I'm not sure about is how you ended up with invalid data in /etc/bogons, do you have the contents of that file from the time? That's not loosely validated like the URL Table aliases were. It has to fetch the list and its md5 via HTTPS, with certificate verification, and if the md5 doesn't match it doesn't put the file into place. Server-side validation also ensures the file's contents are good.

Actions #5

Updated by Alex Vergilis over 8 years ago

Unfortunately, I do not have a sample I can provide. The file contents have been overwritten with other valid downloaded data after the failures.

Is there an md5 validity check for URL Table aliases that can be implemented to minimize errors?

Actions #6

Updated by Chris Buechler over 8 years ago

  • Status changed from Feedback to Resolved

This all works.

Alex Vergilis wrote:

Is there an md5 validity check for URL Table aliases that can be implemented to minimize errors?

not at this time. With all the validation here now I don't think it would help, shouldn't be any circumstance where a web server delivers a file with a 200 return code that wouldn't match a md5.

Actions

Also available in: Atom PDF