Maintain IP range tables for popular Internet sites
Current version of pfsense includes the filterdns daemon which periodically resolves any fqdn in aliases into IP. But this won't work for Websites that return a different set of IPs on each DNS request, so the current solution seems to be doing URL filtering via a proxy like Squid+squidhuard. However this is of little help when a company has moved their email to Google and needs to access its servers via IMAP and wants to whitelist all Google's IPs. This scenario will come up more often, as companies migrate into SaaS and the cloud.
A solution would be for pfsense to automatically keep track of certain sites' IP ranges (e.g. GoogleApps). This info can be obtained via whois or DNS.
E.g. Google's ASN is 15169 https://www.dan.me.uk/bgplookup?asn=15169 or via DNS lookup of the SPF record, as Google suggests in "Google IP address ranges" page http://www.google.com/support/a/bin/answer.py?answer=60764
$ host -t txt _spf.google.com
_spf.google.com descriptive text "v=spf1 ip4:184.108.40.206/19 ip4:220.127.116.11/19 ip4:18.104.22.168/20 ip4:22.214.171.124/18 ip4:126.96.36.199/17 ip4:188.8.131.52/20 ip4:184.108.40.206/16 ip4:220.127.116.11/20 ip4:18.104.22.168/20 ip4:22.214.171.124/16 ?all"
#1 Updated by Dim Hatz over 9 years ago
A somewhat related recent discussion http://forum.pfsense.org/index.php/topic,38741.0.html
#2 Updated by Seth Mos over 9 years ago
For this to be useful or "complete" it would need a hybrid approach. You can't be sure that the spf record will also give you access to the google search and apps platforms.
I frequently run into this as well.
The idea of allowing a ASN number for firewall rules is tempting though, the easiest way to access this in a reasonably light fashion is by consulting a route-server that can give you all the networks.
This includes v6 and would make it worthwile. Pretty sure that route-servers have policies forbidding you to use it. I'd look into a package to see if that can work.
.. more thoughts. We could dump the ASN numbers to plain txt files to be picked up off a pfsense server or mirror. This could be done on a daily or hourly basis, as long as we have access to a route-server.
Another thought that just occured is that this would work really well for google, but not quite so well for other non content networks. Granular it isn't
#3 Updated by Dim Hatz over 9 years ago
"We could dump the ASN numbers to plain txt files to be picked up off a pfsense server or mirror"
Right, that would be another viable option, much like the /etc/rc.update_bogons.sh does for bogons.
Actually there are a number of txt files containing IP ranges in CIDR notation that could be added to pfSense (and I've seen them being used in other software firewalls) with scripts very much like the rc.update_bogons.sh-script, e.g. the lists by dshield.org or spamhaus.org e.g.
DROP (Don't Route Or Peer) is an advisory "drop all traffic" list, consisting of stolen 'hijacked' netblocks and netblocks controlled entirely by professional spammers. DROP is a tiny subset of the SBL designed for use by firewalls and routing equipment.
DShield's current Most Active Attacking IPs
(Same data as is used on DShield.org Top 10 Most Wanted.)
0 = IP Address, 1 = Resolved domain of IP Address
And such a targeted approach is much better than blocking entire countries, as some people do.
#4 Updated by Jim Pingle over 9 years ago
Lists like that can be added as URL table aliases in 2.0 (though the code may need adjusting so it only grabs the first parameter of the line in those format files). URL Table Aliases are aliases pointed at an arbitrary URL that contains a text list of CIDR networks, then you can use that alias in firewall rules however you like. The list is updated periodically.
#6 Updated by Dim Hatz over 9 years ago
Sure if you would want to "cover all the bases" you'd need to monitor DNS traffic and add IPs to the passthrough list, however for starters it could be something much simpler, like an enhanced version of
/etc/rc.update_urltables with the changes that jimp mentioned (BTW note that the "Update Freq." option is inactive in firewall_aliases_edit.php in 2.0REL).