Project

General

Profile

Bug #5848

Downloaded rules data validation

Added by Alex Vergilis almost 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Category:
Rules / NAT
Target version:
Start date:
02/06/2016
Due date:
% Done:

0%

Estimated time:
Affected Version:
All
Affected Architecture:

Description

Occasionally, data is not properly downloaded from internet based sources, and rules cannot be generated with errors similar to:

php-fpm[]: /rc.filter_configure_sync: New alert found: There were error(s) loading the rules: /tmp/rules.debug:19: file "/etc/bogons" contains bad data - The line in question reads [19]: table <bogons> persist file "/etc/bogons" 
php-fpm[]: /rc.filter_configure_sync: New alert found: There were error(s) loading the rules: /tmp/rules.debug:202: syntax error - The line in question reads [X]: pass  in  quick  on $IPsec inet proto tcp  from $Network1 to $Network2 port $Ports tracker 123 flags S/SA keep state  label "USER_RULE" 
  • If there are errors downloading the data, configurable attempts should be made to retry until successful data downloads or the number of attempts is exhausted. (with a configurable pause between attempts)
  • If data is not in the expected format, for ports or cidr blocks, it should not be saved. (i.e. a 404/503 error text)
  • If the rule points to an empty (or invalid) data file, it should not be loaded.

Associated revisions

Revision fd4dbabc (diff)
Added by Chris Buechler almost 4 years ago

Return false in download_file rather than the failed status code when a download fails. Return would always evaluate to true previously though other parts of the code expect a false value when a download fails. related to Ticket #5848

Revision d280ec9b (diff)
Added by Chris Buechler almost 4 years ago

Don't try to process URL alias if the download fails. Ticket #5848

Revision 37af5cf5 (diff)
Added by Chris Buechler almost 4 years ago

Don't try to process aliases whose downloads have failed in update_alias_url_data(). Ticket #5848

Revision ca46f1de (diff)
Added by Chris Buechler almost 4 years ago

Return false if download fails in process_alias_urltable so input validation based upon it works. Ticket #5848

Revision 287df967 (diff)
Added by Chris Buechler almost 4 years ago

Show user the URL causing the error. Ticket #5848

Revision e9fea9dc (diff)
Added by Chris Buechler almost 4 years ago

Parse URL Table alias downloads with parse_aliases_file to ensure only valid contents. Ticket #5848

Revision b913daf8 (diff)
Added by Chris Buechler almost 4 years ago

add an option to only validate URL tables, so stray files aren't left behind when input validation fails. Ticket #5848

Revision cc293ac0 (diff)
Added by Chris Buechler almost 4 years ago

If URL table file size is 0, force update. Ticket #5848

History

#1 Updated by Jim Thompson almost 4 years ago

  • Status changed from New to Feedback
  • Assignee set to Chris Buechler

What "Internet sources"?

#2 Updated by Alex Vergilis almost 4 years ago

RE "internet sources"?

From the 2 provided log samples of failures that occur across all of my tracked systems:

1 - Bogon Networks

Interfaces > Interface > Block bogon networks

Loaded by cron, for example: /etc/rc.update_bogons.sh

2 - Aliases

Firewall > Aliases > URL (IPs)
Firewall > Aliases > URL (Ports)
Firewall > Aliases > URL Table (IPs)
Firewall > Aliases > URL Table (Ports)

Loaded by cron, for example: /etc/rc.update_urltables

#3 Updated by Chris Buechler almost 4 years ago

  • Status changed from Feedback to Confirmed
  • Affected Version changed from 2.2.5 to All

#4 Updated by Chris Buechler almost 4 years ago

  • Status changed from Confirmed to Feedback

I've been through a slew of circumstances that were broken before and are all fine now. There are multiple levels of additional verification that prevent invalid data from getting into a table or the ruleset.

The only part I'm not sure about is how you ended up with invalid data in /etc/bogons, do you have the contents of that file from the time? That's not loosely validated like the URL Table aliases were. It has to fetch the list and its md5 via HTTPS, with certificate verification, and if the md5 doesn't match it doesn't put the file into place. Server-side validation also ensures the file's contents are good.

#5 Updated by Alex Vergilis almost 4 years ago

Unfortunately, I do not have a sample I can provide. The file contents have been overwritten with other valid downloaded data after the failures.

Is there an md5 validity check for URL Table aliases that can be implemented to minimize errors?

#6 Updated by Chris Buechler almost 4 years ago

  • Status changed from Feedback to Resolved

This all works.

Alex Vergilis wrote:

Is there an md5 validity check for URL Table aliases that can be implemented to minimize errors?

not at this time. With all the validation here now I don't think it would help, shouldn't be any circumstance where a web server delivers a file with a 200 return code that wouldn't match a md5.

Also available in: Atom PDF