pfSense

I believe my issues may be related to this. We updated to 2.4.4 p2 on Jan 9, but only in the past few days have seen the problems.

The firewalls and sites at several locations are configured to allow remote access based on firewall aliases and rules using those aliases. The aliases sometimes contain a mix of IP addresses (/32), IP networks (/29 for ex), and DNS names (something.company.com).

Since the past few days, the pfSense firewalls at the various sites all reject my remote connection attempts, and the rejections are visible in the firewall logs.

It's...a problem.

2.4.4-RELEASE-p2, I've had this multiple times. At the moment I can even sort of reproduce it.
When adding hosts to an alias my AD DNS server logs:

2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Rcv <pfsense IP> a463 Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Rcv <pfsense IP> 519a Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Snd <pfsense IP> a463 R Q [8081 DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Snd <pfsense IP> 519a R Q [8085 A DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

This is an external host, i.e. a DNS that needs to be externaly resolved by our DNS servers. That seems to work fine result gets send back to pfSense. However the host does NOT end up in the table for that alias. When I add another DNS, same domain, so hosted at the same DNS on internet, that works fine. I tried others like www.tweakers.net, www.nos.nl or bbc.co.uk I have the same success loggings in my DNS debug log, and they DO end up in the alias table as well.

pfSense Resolver log:
Feb 18 12:47:14 filterdns Adding host <Host that gets added to the alias> (I just added that one to the alias)
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <Host that gets added to the alias>
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <host that does NOT end up in table> (I just added that one in the alias as well)
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <existing host, which was already in the alias>

The host that does NOT end up in table here, is by the way successfully added to some other aliasses, where it works just as expected. But for this alias I am missing the 'Adding host' in the pfSense log.

I tried creating a new alias, with the same three hosts as in the alias I used above. Here NONE of them end up in the table, after waiting for about 20 minutes, while in the alias used above two out of three (and the same two every time, no matter what order I put them in) work. Then I added www.tweakers.net as another try, and that one gets in there immediately.
I again killed filterdns, restarted it and poof - the tables immediately got filled as they should. So it seems filterdns is partially functional - some hosts get added, some aren't. It could indeed be when hosts already exist in the table somewhere; however restarting filterdns at least populates them for a while.

Tell me what loggings you need. As it seems I can now reproduce this at will (also on my second carp / HA node by the way) I can probably give all needed logs.

I've just downgraded a test-machine to 2.4.4 release, and that works fine. Keeping it there for a while.

Shortly after I posted my problem above 20 days ago, it started working again on its own.

Then today, it is again not working.

So it may be a sporadic issue with the alias resolution, that doesn't happen consistently. Have not been able to pin down the issue at all.

I can confirm my issue is the same as described by the other posters on this bug.

Logs show that filterdns claims to be doing the right thing - all expected alias entries (FQDN's, IP's, networks) show up in:
$ clog /var/log/resolver.log | grep "Adding Action"

But the alias table is incomplete, some IP addresses are missing:
$ pfctl -T show -t my_alias_name

There is no DNS resolution issue with any of the FQDN's - if I ping the FQDN's from the firewall their IP addresses are resolved.

Restarting the filter, re-saving the alias does not help.

I've also ruled out some other possibilities below -

Not the issue:
https://docs.netgate.com/pfsense/en/latest/firewall/thread-error-using-many-hostname-in-aliases.html
(I don't have a threads error in logs, and setting this tunable did not help)

Not the issue:
Mixing FQDN's and IP's - I tried creating a new alias with only a single FQDN from the ones that don't work in the original alias. Still no luck.

I see this behavior on 2.4.4_p2, on 2.4.5-dev and on 2.5.0-dev.
As workaround we can:
- in console run 'pkill filterdns' command
- then /Status/Filter Reload to start 'filterdns' service

As a workaround I have installed the Cron package with the following additional entries:

*/15 * * * * root killall -9 filterdns; sleep 2; /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1     
@reboot      root sleep 10; killall -9 filterdns; sleep 2; /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1

I know it's targeted for 2.5.0, but still I'd like to inform people here that 2.4.4_3 does indeed NOT fix this, making it yet another update that kills main functionality. Be aware, and be extremely reluctant to update!

Setting "Aliases Hostnames Resolve Interval" to 30 seconds (which should be the minimum value) in System/Advanced/Firewall&NAT seems to work around the issue for me (which could be some kind of race condition in filterdns but that's just guessing on my side).

Rudolf Mayerhofer wrote:

Setting "Aliases Hostnames Resolve Interval" to 30 seconds (which should be the minimum value) in System/Advanced/Firewall&NAT seems to work around the issue for me (which could be some kind of race condition in filterdns but that's just guessing on my side).

Based on what Rudolf wrote, I changed the value of "Aliases Hostnames Resolve Interval" from empty to 300secs. That setting combined with a filter reload from "Status/Filter Reload" menu made my rules work as expected again.

I will continue testing and update this post if I find anything else.

Christoforos Tsoukaris wrote:

Based on what Rudolf wrote, I changed the value of "Aliases Hostnames Resolve Interval" from empty to 300secs. That setting combined with a filter reload from "Status/Filter Reload" menu made my rules work as expected again.

I will continue testing and update this post if I find anything else.

If you look at the cron entries I have mentioned earlier (#9296#note-11), you will see that I have the interval (-i) set to 300. I am still seeing missing entries in the alias tables on occasion, which do get corrected when cron kills and restarts filterdns within the next 15 mins.

The crontab entries as mentioned in #11 didn't run as they were just keeping on adding new filterdns processes, eventually causing the firewall to trigger CARP/HA, give high latency to VPN and internet traffic, and eventually cause the firewall to stop passing traffic altogether. I went for installing the cron GUI package (But you could just as easily edit /etc/crontab directly), and I've changed the lines to:

*/15 * * * *  root    /usr/bin/pkill -f "/usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1"; sleep 2; /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1
@reboot       root    sleep 10; /usr/bin/pkill -f "/usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1"; sleep 2; /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1

I've checked that these cron jobs now correctly kill the old processes before starting new ones, and that the filterdns process is correctly doing it's job as long as the restarts are in place (As noted in other notes to this bug, it still stops resolving after a while, although I've not had time to monitor the firewalls to find out exactly how long it is before the process stops, so apologies for that).

Mark Monaghan wrote:

The crontab entries as mentioned in #11 didn't run as they were just keeping on adding new filterdns processes,

Very interesting. I'm not seeing that occur at all (and it was something I was monitoring closely when I set it up). I wonder what makes this operation different on your system ? Could you have possibly forgotten the "-9" argument to "killall" ?

Gavin Stewart wrote:

Very interesting. I'm not seeing that occur at all (and it was something I was monitoring closely when I set it up). I wonder what makes this operation different on your system ? Could you have possibly forgotten the "-9" argument to "killall" ?

No, sorry, I copied and pasted the commands verbatim from here to ensure that I didn't make any errors when implementing them. The -9 was definitely in there. What I was finding was that if I ran them individually, or even as a grouped command set, from the CLI, they worked perfectly, but they failed to kill any processes when ran via the cron. This was all done and tested on 2.4.4-p3. I cannot comment on how this performed prior to this version, as it wasn't implemented on 2.4.4-p2 or lower, only after the system was upgraded to run on the latest stable version.

This is the reason I switched the cron job to pkill as nothing I tried would get killall or even kill to terminate the filterdns process via the cron, but pkill was working for reasons only known to the OS. However, that also presented it's own challenges as unless I used the exact filter of -f "/usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1" rather than pkill filterdns or pkill -f "filterdns" it was seeing filterdns in it's own cron job and killing itself before it killed any filterdns processes, so until I put the large filter in place, I wasn't any further forward and still had filterdns processes stacking up, causing the system to fall over eventually (I put this down to the system running out of open file handles as it certainly never ran out of memory before the crashes that I experienced started to happen.

Rudolf Mayerhofer wrote:

Setting "Aliases Hostnames Resolve Interval" to 30 seconds (which should be the minimum value) in System/Advanced/Firewall&NAT seems to work around the issue for me (which could be some kind of race condition in filterdns but that's just guessing on my side).

As a follow up: With 30 seconds resolve interval things are still working fine one month later without killing/restarting filterdns.

Rudolf Mayerhofer wrote:

As a follow up: With 30 seconds resolve interval things are still working fine one month later without killing/restarting filterdns.

Unfortunately this doesn't solve my problem. Same issues, regardless of the refresh value. Continues to make life difficult on a regular basis.

I think I have a similar problem.
My inbound rule did not work with an FQDN in the Alias. (Whitelist for source addresses) Weird thing was that only one ip-address in the Alias (not the FQDN) did not work, restarting servers/pfsense and such did not give any result.

When I enable the 300 sec for 'Aliases Hostname Resolve interval', it WORKS, when i leave it empty, it FAILS directly.

2.4.4-RELEASE-p3, FreeBSD 11.2-RELEASE-p10

Netgate SG-4860 running 2.4.4-RELEASE-p3 (amd64). At least twice I've experienced issues, I assume involving filterdns, where aliases were updated via the GUI but did not make it into the pf table (and I believe the updates didn't land in /var/etc/dnsfilter.conf either, but I can't confirm).

The alias/table is 58 entries, mostly DNS names, some IP addresses, and some other pfSense aliases. The last time this problem happened neither DNS name, IP address, nor pfSense aliases were added to the pf table. filterdns was running. killall -9 filterdns && /etc/rc.filter_configure fixed it.

Also experiencing this issue on 2.4.4-p2 and now 2.4.4-p3. If FQDNs are remove the table updates correctly. Due to the way many cloud service provider clusters move and reallocate IPs it is impractical to use individual host or CIDR lists as this requires constant updating once a change is made on the remote server. In my case I first noted the issue after one entry was updated and became a duplicate IP in a CIDR that was in the table. I've tried killing and restarting filterdns and changing resolution time to 30 seconds but with no improvement. Sometimes the table would partially generate, other times it was completely empty.
Can this be escalated for a resolution before 2.5 as it breaks the core firewall functionality for those of us using a cloud or hybrid cloud setup?

I second Justins message / question. pfSense is completely unusable after 2.4.4 initial release. With filterdns not working properly, it fails as a firewall completely. That leaves us with the choice of updating pfSense to a completely useless not working version, or leave it at 2.4.4 and leave it vulnerable to now known security issues. Both are unacceptable.

This should certainly get higher priority.

Most of you are more experienced at this than me so please be tolerant if this is a dumb question.

I added a Firewall Alias containing a single FQDN, xxx.mydomain.com. A Diagnostics=>DNS Lookup for xxx.mydomain.com returns three addresses, one IPv4 and two IPv6 (received from DHCPv6 and RA). However when I inspect the alias's table using Diagnostics=>Tables, only the IPv4 address is listed. It follows that the rule referencing the alias isn't passing traffic to the IPv6 addresses. I would expect to see all three addresses in the table. Is that expectation correct or am I missing something?

For background, I am using resolver and there is a domain override in place for mydomain.com (TypeTransparent).

Am I experiencing this bug?

That sounds like it might be something else Tom. Check your output from the CLI with: pfctl -T show -t ALIASNAME
If it's not there try making a forum post as the discussion here should be directly about the bug, not diagnosing the possibility of your setup having it.

Justin J: I took your advice and posted on the forum and was promptly referred back here. Here's the link in case you are inclined to read it. https://forum.netgate.com/topic/145553/firewall-alias-not-updating-table-correctly

It's been about 8 months now that we are unable to update / patch our firewalls because of this. Yeah I know, open source, contribute yourself if you don't like it, and so on. But still no update on this from pfSense team? By now we are seriously considering moving away. It's unacceptable that we have to run a firewall that's by now simply not secure anymore because of the now public security flaws.

Can we get an official status update on this? And why this doesn't get higher priority? Is there an official ETA for 2.5?

Robert Gijsen wrote:

It's been about 8 months now that we are unable to update / patch our firewalls because of this. [...] And why this doesn't get higher priority? Is there an official ETA for 2.5?

This issue is becoming a show stopper for us as well.

Ph. T wrote:

If you are using FQDN-Aliases each FQDN can only be used once, if
you use the alias twice, the generated tables are incomplete.

No DNS-Server/Resolver on the firewall is used. External DNS
resolvers are configured.

Example:
alias1 : fqdn1, fqdn2, fqdn3
alias2 : fqdn4, fqdn2, fqdn3

Generated tables are incomplete
alias1 : fqdn1, fqdn2, fqdn3
alias2 : fqdn4 (the others are missing)

alias2 does only contain fqdn4 and fqdn2 and fqdn2 are missing.

This bug seems to arise with 2.4.4_p1 and is still existing in 2.4.4_p2;
I am not sure if this behavior is present within 2.4.4.

I am working on a minimal example which i will provide.

I had the same problem. The rules stack has a limit and this means that domain names cannot be resolved. For example:

Alias-1 => Network => "10.10.0.0/24" It is not equal to "10.10.0.0-10.10.0.254"

While 10.10.0.0/24 is a valid nomenclature for a rule, but 10.10.0.0-10.10.0.254 say that the entire range should be described:
10.10.0.0
10.10.0.1
10.10.0.2
10.10.0.3
10.10.0.4
..
...
....
10.10.0.254

This range of IPs causes a problem for the "next" aliases in the system, and it is very possible that they cannot be resolved.

I have removed all gigantic ranges of IPs and the problem is solved.

This problem has already affected several cloud providers, so they mostly do not accept using FQDN aliases in their routing rules or ACLs.

Angel Briceño wrote:

I have removed all gigantic ranges of IPs and the problem is solved.

I have no ranges of IP addresses (only networks defined in CIDR notation), and the problem persists.

I am very,very unhappy with the time it takes to deal and fix this problem.
Is there any way to speed up the process ? I can provide any additional info.

I think Angel has a complete different problem. The limit of table entries
has been an issue due to big bogon tables some time ago. A fix might be to
increase the maximum table entries, or not to use the bogon table.

    System > Advanced >Firewall & NAT
    Firewall Maximum Table Entries 

Ph. T wrote:

I am very,very unhappy with the time it takes to deal and fix this problem.
Is there any way to speed up the process ? I can provide any additional info.

I think Angel has a complete different problem. The limit of table entries
has been an issue due to big bogon tables some time ago. A fix might be to
increase the maximum table entries, or not to use the bogon table.
[...]
we have set this value to 400000.

Please, provide the filterdns logs for your case, with debug enabled (-d20).

As strange as it may seem, this is proving to be difficult to reproduce reliably.

If you want to send the logs privately, please send it to luiz at netgate.com

Thanks.

If anyone can come up with simple cases that reliably reproduce the problem, that would definitely help. That is, the smallest possible configuration that results in the problem happening. For example, an alias and firewall rule which exhibit the problem (or multiple aliases+rules) along with whatever other conditions are necessary, such as waiting specific amounts of time, or having invalid hostnames in the alias, etc. Along with log data mentioned above and the contents of /var/etc/filterdns.conf.

I will provide the data / config.xml . I could also provide a virtual-box pfsense-installation
which shows this problem. I hope i could provide this today.

I see similar effects with the old config which i attached in January.

Jim Pingle wrote:

If anyone can come up with simple cases that reliably reproduce the problem [...]

What's the status here? Has Netgate been able to reproduce this issue?

John K wrote:

What's the status here? Has Netgate been able to reproduce this issue?

Not that I have seen yet. We still need to find a combination of settings that reliably and repeatedly reproduces the issue.

Jim Pingle wrote:

John K wrote:

What's the status here? Has Netgate been able to reproduce this issue?

Not that I have seen yet. We still need to find a combination of settings that reliably and repeatedly reproduces the issue.

I just installed a brand new pfsense box and once I created an alias with an FQDN it didn't work, when I checked it on TABLES I only had the entries' list above the fqdn entry, the fqdn itself and everything else were missing.

Once I deleted the FQDN entry and hit filter reload, the alias was correct.

Vinicius DellAglio wrote:

I just installed a brand new pfsense box and once I created an alias with an FQDN it didn't work, when I checked it on TABLES I only had the entries' list above the fqdn entry, the fqdn itself and everything else were missing.

Once I deleted the FQDN entry and hit filter reload, the alias was correct.

Is the FQDN a CNAME, A record, etc?

Jim Pingle wrote:

John K wrote:

What's the status here? Has Netgate been able to reproduce this issue?

Not that I have seen yet. We still need to find a combination of settings that reliably and repeatedly reproduces the issue.

I now have a minimal and repeatable set of steps to reproduce this.

• New VM for FreeBSD 64-bit, accepting all defaults
• First network adapter stays NAT (used as WAN in pfSense)
• Add second network adapter as host-only (used as LAN in pfSense) to access web interface from host
• pfSense-CE-2.4.4-RELEASE-p3-amd64.iso.gz, accepting all defaults
• configure LAN IP address as needed at pfSense console to match host-only network settings

1. Firewall -> Aliases -> Import
2. Alias Name "TEST1", import entire alias list below, Save
3. Diagnostics -> Tables -> "TEST1"
   + Note approx 59 entries, specifically the 192.168.x.x addresses.
   + (It may take a couple of page reloads for all the addresses to resolve.)
4. Firewall -> Aliases -> Import
5. Alias Name "TEST2", import entire alias list below, Save
6. Diagnostics -> Tables -> "TEST2"
   + Note that table is never fully populated (even if waiting longer than the 5 minute filterdns interval).
   + Note that 192.168.x.x addresses do not all appear.
7. Kill the filterfdns process at the console, and restart manually:
   /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
   + Note that "TEST2" table is now properly populated.

List of aliases to import:

mail.bigpond.com
speedtest.telstra.com
mirror.internode.on.net
192.168.1.1
mirror.aarnet.edu.au
account.mojang.com
authserver.mojang.com
192.168.2.2
mc.hypixel.net
sessionserver.mojang.com
launchermeta.mojang.com
192.168.3.3
apps.yourtown.com.au
obook4.oxforddigital.com.au
www.oxforddigital.com.au
192.168.4.4
www.distance.vic.edu.au
lms.decvonline.vic.edu.au
connect.vic.edu.au
192.168.5.5
us.mineplex.com
stileapp.com
stileeducation.com
192.168.6.6

Jim Pingle wrote:

John K wrote:

What's the status here? Has Netgate been able to reproduce this issue?

Not that I have seen yet. We still need to find a combination of settings that reliably and repeatedly reproduces the issue.

Given Gavin Stewart's steps above, has Netgate finally been able to reproduce this issue now?

Gavin Stewart wrote:

I now have a minimal and repeatable set of steps to reproduce this.

Actually, I have revised the list of aliases to just one single host, still repeatable:

mail.bigpond.com

As per my previous instructions, with a clean install, and adding two host aliases for mail.bigpond.com (TEST1 and TEST2, applying changes between them), this is the entire clog output from /var/log/resolver.log (filterdns with -d 20):

Nov 27 06:18:13 pfSense filterdns:  Adding Action: pf table: TEST1 host: mail.bigpond.com
Nov 27 06:18:13 pfSense filterdns:      Adding host mail.bigpond.com
Nov 27 06:18:13 pfSense filterdns: Creating a new thread for action type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:18:13 pfSense filterdns: Creating a new thread for host mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns:      found address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns:          adding address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns: Change detected on host: mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns:  Awaking from the sleep for type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns:      Added pf address, table: TEST1 host: mail.bigpond.com address: 203.36.137.241
Nov 27 06:18:20 pfSense filterdns:  Updated pf table TEST1 host: mail.bigpond.com error: 0
Nov 27 06:18:24 pfSense filterdns: Received signal Hangup(1).
Nov 27 06:18:24 pfSense filterdns: merge_config: configuration reload
Nov 27 06:18:24 pfSense filterdns: Copied 1 actions to old
Nov 27 06:18:24 pfSense filterdns:  Adding Action: pf table: TEST1 host: mail.bigpond.com
Nov 27 06:18:24 pfSense filterdns:  Adding Action: pf table: TEST2 host: mail.bigpond.com
Nov 27 06:18:24 pfSense filterdns: Copied 2 actions to new
Nov 27 06:18:24 pfSense filterdns: Cleaning up action type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:18:24 pfSense filterdns: Loaded actions: 1 old and 1 new = 2 total
Nov 27 06:18:24 pfSense filterdns: Cleaning up previous actions
Nov 27 06:18:24 pfSense filterdns: Creating a new thread for action type: pf table: TEST2 hostname: mail.bigpond.com
Nov 27 06:18:24 pfSense filterdns:  Awaking from the sleep for hostname mail.bigpond.com (2)
Nov 27 06:18:24 pfSense filterdns:      found address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:23:23 pfSense filterdns:  Awaking from the sleep for hostname mail.bigpond.com (2)
Nov 27 06:23:24 pfSense filterdns:      found address 203.36.137.241 for host mail.bigpond.com

[2.4.4-RELEASE][root@pfSense.localdomain]/root: cat /var/etc/filterdns.conf
pf mail.bigpond.com TEST1
pf mail.bigpond.com TEST2
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -T show -t TEST1
   203.36.137.241
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -T show -t TEST2
[2.4.4-RELEASE][root@pfSense.localdomain]/root: 

As can be seen, filterdns doesn't ever add the resolved address to the second table, unless filterdns is killed and restarted, resulting in this addition to the log:

Nov 27 06:26:10 pfSense filterdns: Received signal Terminated(15).
Nov 27 06:26:10 pfSense filterdns: Cleaning up action type: pf table: TEST2 hostname: mail.bigpond.com
Nov 27 06:26:10 pfSense filterdns: Cleaning up action type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:26:10 pfSense filterdns: Waiting 2 seconds for threads to finish
Nov 27 06:26:10 pfSense filterdns:  Awaking from the sleep for hostname mail.bigpond.com (0)
Nov 27 06:26:10 pfSense filterdns: Cleaning up hostname mail.bigpond.com
Nov 27 06:26:10 pfSense filterdns:          removing address 203.36.137.241 from host mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns:  Adding Action: pf table: TEST1 host: mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns:      Adding host mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns:  Adding Action: pf table: TEST2 host: mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns: Creating a new thread for action type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns: Creating a new thread for action type: pf table: TEST2 hostname: mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns: Creating a new thread for host mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:      found address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:          adding address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns: Change detected on host: mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:  Awaking from the sleep for type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:      Added pf address, table: TEST1 host: mail.bigpond.com address: 203.36.137.241
Nov 27 06:26:36 pfSense filterdns:  Updated pf table TEST1 host: mail.bigpond.com error: 0
Nov 27 06:26:36 pfSense filterdns:  Awaking from the sleep for type: pf table: TEST2 hostname: mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:      Added pf address, table: TEST2 host: mail.bigpond.com address: 203.36.137.241
Nov 27 06:26:36 pfSense filterdns:  Updated pf table TEST2 host: mail.bigpond.com error: 0

[2.4.4-RELEASE][root@pfSense.localdomain]/root: cat /var/etc/filterdns.conf
pf mail.bigpond.com TEST1
pf mail.bigpond.com TEST2
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -T show -t TEST1
   203.36.137.241
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -T show -t TEST2
   203.36.137.241
[2.4.4-RELEASE][root@pfSense.localdomain]/root: 

I have a fix for this, and have created a pull request.

https://github.com/pfsense/FreeBSD-ports/pull/714

Luiz Souza wrote:

A fix based on Gavin's PR was committed, please let me know if the problem persists.

Thanks

Maybe a stupic question, but as I don't have any git or build tools available within pfSense obviously, how can we test this? Does that mean we'd need to install the 2.5 nightly? Sidequestion, is there any ETA on 2.5 RTM?

• Robert Gijsen wrote:

Maybe a stupic question, but as I don't have any git or build tools available within pfSense obviously, how can we test this? Does that mean we'd need to install the 2.5 nightly? Sidequestion, is there any ETA on 2.5 RTM?

In a (very large) nutshell:

$ git clone https://github.com/pfsense/freebsd-ports pfsense-ports
$ git clone -b RELENG_2_4_4 https://github.com/pfsense/freebsd-src pfsense-src
# poudriere ports -cp pfsense -m null -M $PWD/pfsense-ports
# poudriere jail -cj pfsense244 -m src=$PWD/pfsense-src -b
# echo ALLOW_UNSUPPORTED_SYSTEM=1 >> /usr/local/etc/poudriere.d/pfsense-make.conf
# poudriere bulk -j pfsense244 -p pfsense -z default net/filterdns
$ scp /usr/local/poudriere/data/packages/pfsense244-pfsense-default/All/filterdns-2.0_3.txz $WHEREVER
$ ssh $WHEREVER pkg install -f filterdns-2.0_3.txz

The ALLOW_UNSUPPORTED_SYSTEM line is necessary if the next line fails (on a FreeBSD 12 build system).

• Luiz Souza wrote:

A fix based on Gavin's PR was committed, please let me know if the problem persists.

Confirmed. With rebuilt filterdns-2.0_3 on pfSense 2.4.4-p3, the tables are now populated correctly.

Luiz Souza wrote:

A fix based on Gavin's PR was committed, please let me know if the problem persists.

Thanks

I have compiled the package for our test-environment (huge thanks to Christian Ullrich for the info, I couldn't have done that without his help) and so far tables are populated as it should now.

Jim Pingle wrote:

Needs checked and/or tested again on 2.4.5 snapshots

tested on pfSense 2.4.5.a.20191220.1407

works as expected,
Resolved

Great to hear about the fix! Would have loved to see a 2.4.4 update with this fixed package, or even just a fixed filterdns package I could download and install on all my 2.4.4 setups, rather than waiting an unknown and growing number of months for 2.4.5 to be released.

As it is I'm spending a number of hours downloading and setting up a FreeBSD 11.2 virtual machine, installing git/poudriere, wastefully downloading multiple GB's of ports and source, and going through the filterdns package rebuild process. And hoping I'll have a fixed filterdns package at the end of this process.

This issue has been an ongoing pain for over a year and even trying to apply the fix is no carnival. But thanks again to the folks who actually dug into and resolved the issue.

It appears a reboot was required on each firewall after updating the filterdns package to my custom built one (2.0_3). Without the reboot, the alias table was still not fully and correctly populated (i.e. there were still ip addresses missing from the list when running $ pfctl -T show -t my_alias_name). Hopefully the problem is well and truly solved now dammit :).

Still not working properly, at least a couple of IP's are still not populating in the table. Giving up for now, will wait for 2.4.5 and see if that somehow fixes things. Maybe my filterdns build didn't pick up the latest code fix or something, no idea.

Eduard Rozenberg wrote:

Still not working properly, at least a couple of IP's are still not populating in the table. Giving up for now, will wait for 2.4.5 and see if that somehow fixes things. Maybe my filterdns build didn't pick up the latest code fix or something, no idea.

Version 2.4.5 already has several RCs. I think it is safer to upgrade to the RC than to do such a process. At least in this case, due to the urgency. The final version should come out in a few days or maybe hours. I have three days using the RC and it works very well.

Fabián Burbano wrote:

Version 2.4.5 already has several RCs. I think it is safer to upgrade to the RC than to do such a process. At least in this case, due to the urgency. The final version should come out in a few days or maybe hours. I have three days using the RC and it works very well.

Thanks! My problem is definitely NOT resolved by my filterdns custom build. Having exactly the same issue. Will upgrade to 2.4.5 when it releases, hopefully soon. Got burned by at least 2-3 pfSense version upgrades in the past, so would like to minimize risk by waiting for 2.4.5 official release.

I also have to come back to my conclusion it was ok with the rebuild filterdns. While working better than before, tables are still not filled correctly when a FQDN is used multiple times in the aliasses.

This bug still persists in Build 2.4.5.r.20200307.0900.
.

This bug still persists in Build 2.4.5 date:2020.04.09

How it was tested ? What was the result ? How it failed ?

This bug still persists in Build 2.4.5 date:2020.04.10

I can confirm my issue is the same as described by the other posters on this bug.

Logs show that filterdns claims to be doing the right thing - all expected alias entries (FQDN's, IP's, networks) show up in:
$ clog /var/log/resolver.log | grep "Adding Action"

But the alias table is incomplete, some (FQDN) addresses are missing:
$ pfctl -T show -t my_alias_name

There is no DNS resolution issue with any of the FQDN's - if I ping the FQDN's from the firewall their IP addresses are resolved.

Restarting the filter, re-saving the alias does not help

Yep. Same issue. Today got locked out again out of all our sites. My workaround is to use a personal VPN to force my own IP to change, which then normally will allow me back in once the firewall re-evaluates the DNS name -> IP mapping in the hostnames alias. As others have speculated, issue may be linked to cases where there are multiple hostnames which evaluate to the same IP inside a pfSense alias. But I don't really have a firm idea or clue.

Same problem here - 2.4.5-RELEASE (amd64)

pfctl -T show -t my_remote
   71.237.XX.XX
   97.120.XX.XX
   184.100.XX.XX

cat /var/etc/filterdns.conf 
pf connection1.dyndns.org my_remote
pf connection2.dyndns.org my_remote
pf connection3.dyndns.org my_remote

My favorite part is the firewall log, showing the blocks.

    May 30 13:42:37    COMCAST      71.237.XX.XX:30614      96.65.XX.XX:21443    TCP:S

Wish I knew how to fix it.. the "Aliases Hostnames Resolve Interval" didn't help :(

Donn Lasher wrote:

Same problem here - 2.4.5-RELEASE (amd64)

This is confirmed.

I am able to replicate the failure in a test VM, using my instructions in #9296#note-44

I will point out that I cannot replicate the failure with just one host alias as described in #9296#note-46

Gavin Stewart wrote:

This is confirmed.

I am able to replicate the failure in a test VM, using my instructions in #9296#note-44

Just revising this statement, because the error isn't immediately reproducible with the instructions linked above. There is a distinct change in the symptoms, and it now appears to only relate to IP addresses in aliases, and not FQDNs.

I now have the following steps to reproducibly demonstrate failure:

1. Firewall -> Aliases -> Import
2. Alias Name "TEST1", import entire alias list below, Save, Apply Changes
3. Diagnostics -> Tables -> "TEST1"
   + Note 2 entries, specifically the 192.168.1.1 address.
4. Firewall -> Aliases -> Import
5. Alias Name "TEST2", import entire alias list below, Save, Apply Changes
6. Diagnostics -> Tables -> "TEST2"
   + Note table is the same as TEST1
7. Alias Name "TEST3", import entire alias list below, Save, Apply Changes
8. Diagnostics -> Tables -> "TEST3"
   + Note that 192.168.1.1 addresses does not appear.
9. Kill the filterfdns process at the console, and restart manually:
   /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
   + Note that "TEST3" table is now properly populated.

List of aliases to import:

mail.bigpond.com
192.168.1.1

• Test may be replicated over again by deleting all aliases and starting again.
• Test appears to always require third alias to occur
• This case also fails in the patched filterdns I have been using with 2.4.4-RELEASE-p3, so it would appear there is still an undiagnosed bug in filterdns, this ticket should probably be reopened.

I just ran into this today. I was using IP addresses for the bookkeeper. She finally got a hostname with DynDNS.
So I went to Firewall>Aliases and changed her IP address into the hostname. I verified that the hostname resolves to her IP address. She connected once (with RDP) with no problems.
Subsequent tries fail. She gets the RDC dialog that says Remote Desktop can't connect to the remote computer for one of these reasons....
I add the IP back into the alias and she is again able to connect via RDP.
Interesting development here...My hostname works without issue. My hostname is a CNAME I added to my GoDaddy account, pointing to my public static IP.
Her hostname does not work. Her public IP is dynamic, and she uses DynDNS. What's the difference? They both resolve to a public IP correctly.

Thanks for the detailed instructions Gavin.

I pushed a fix which should do the right thing in this case.

Please test the new version (filterdns-2.0_4) and let me know if the problem persists.

• The new package is available with the 2.5 snapshots.

Luiz Souza wrote:

Please test the new version (filterdns-2.0_4) and let me know if the problem persists.

This problem persists unfortunately.

I have recompiled filterdns from a pull of the latest devel branch (55691996), and the problem remains while following the instructions in #9296#note-73

Should the status on this be changed? It says resolved.

Brendon Baumgartner wrote:

Should the status on this be changed? It says resolved.

Definitely not resolved. It's an unpredictably occuring issue that continues to terrorize admins.

There's related strange behavior - using an IP address that was part of an alias, still cannot access the network when I use that IP as part of an allow from source rule.

The only way I can deal with the issue is to use a VPN to change my own IP, and use the VPN IP in the dynamic FQDN specified in the alias.

I just registered here to say that I believe I'm experiencing this exact bug (see https://forum.netgate.com/topic/159856/pass-rule-as-an-exception-to-a-reject-rule-doesn-t-match). If it's indeed the same bug, it should not be considered resolved, because I'm using the latest snapshot (2.5.0.a.20210113.2350) nor should it be considered low priority, because it essentially breaks the firewall functionality of a firewall OS. Both workarounds posted earlier (#9296#note-14 and #9296#note-16) did not work for me. Neither did reinstalling and importing the config.xml.

The only workaround that does actually work is running a script on a webserver that resolves lists of FQDNs and creating URL aliases in pfSense pointing to that script, which seems a bit ridiculous.

However, thanks to the people who are working on resolving this issue!

same issue on 2.5.0.a.20210120.1500

mixed alias entries:
- yandex.ru
- 1.2.3.4

# pfctl -t mix -Ts
   213.180.193.56

but if you create another one alias (any type) and click 'apply':

# pfctl -t mix -Ts
   1.2.3.4
   213.180.193.56

the issue seems to only occur when the alias is first created

Wait wut? This got postponed AGAIN? This is a breaking issue for two years and a few days now, and still it's priority is 'low' and it gets postponed again? How much more info to reproduce do you actually need?

Sorry for my rant, but in all sincerity I, and I assume a lot of us, would like to know by now where this is going. I've been unable to upgrade pfSense for almost 2 years without breaking our setup completely because of this, which makes it an unsafe box however you want to explain it. What can we do more to help you troubleshoot or fix this?

Same sentiment here as Robert Gijsen's above.

Do we at least know whether the bug is in filterdns itself (generating wrong updates) or in pf (processing valid updates wrongly)? If it is filterdns's fault, I think the best way to proceed is to rewrite it. The way it works at the moment is to run one thread per host name and one thread per alias, so any reasonable setup quickly reaches into the hundreds of threads, all mostly waiting.

The last proposed fix (that I initially thought was working) was changing the way all these threads are communicating with each other, which is always the most difficult part to get right.

same issue on 2.5.2-RELEASE - date 20211109

3 years...

I use 2.5.2-RELEASE (amd64) - Jul, 02 15:33:00 EDT 2021 - with exactly the same problem.
After killing filterdns (pkill filterdns) and reloading filter (Status -> Filter Reload) tables are updated correctly, but only for some minutes (probably until automatic refresh occurs), when the same alias table drop number of addresses from eg. 135 to 82. Even after that count of addresses in this example alias table changes between from about 76 up to 90 (when it should has above 130).
I post sample resolver logs here - it seems to have some errors, but my knowledge is too low to resolve this.

Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: watson.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: watson.microsoft.com address: 20.42.73.29
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: watson.microsoft.com address: 13.89.179.12 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: watson.microsoft.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: secure.meetup.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: secure.meetup.com address: 151.101.114.217
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: secure.meetup.com address: 151.101.14.217 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: secure.meetup.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.55 for host shredder.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.21.226 for host ocsp2.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.20.226 for host ocsp2.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:15e2 for host ocsp2.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:14e2 for host ocsp2.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.176 for host officepreviewredir.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.20.226 for host secure.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.21.226 for host secure.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:14e2 for host secure.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:15e2 for host secure.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.42.12 for host d.docs.live.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.121 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.109.88.121 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.83 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.109.88.83 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.46 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.109.88.46 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.120 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.109.88.120 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.109.76.98 from host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.109.76.122 from host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.109.76.123 from host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.109.76.99 from host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.88.120
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.88.46
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.88.83
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.88.121
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.76.98 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.76.122 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.76.123 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.76.99 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: nleditor.osi.office.net error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 51.116.48.199 for host informationprotection.hosting.portal.azure.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 20.82.250.189 for host smartscreen.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 23.197.104.63 for host officecdn.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a02:26f0:7100:1a1::6bb for host officecdn.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a02:26f0:7100:1b4::6bb for host officecdn.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 204.79.197.200 for host c.bing.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.42.12 for host docs.live.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.20.226 for host crl.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.21.226 for host crl.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:15e2 for host crl.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:14e2 for host crl.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 162.125.248.18 for host dropbox.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:100:6040:18::a27d:f812 for host dropbox.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.76.0 for host wikipedia.firstpartyapps.oaspapps.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2.16.172.58 for host isrg.trustid.ocsp.identrust.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 92.123.189.128 for host isrg.trustid.ocsp.identrust.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a02:26f0:64::215:f2c5 for host isrg.trustid.ocsp.identrust.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a02:26f0:64::215:f2bb for host isrg.trustid.ocsp.identrust.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.132 for host protection.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 184.31.81.167 for host videocontent.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.81.113.223 for host apps.skypeassets.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.76.0 for host excelbingmap.firstpartyapps.oaspapps.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 34.252.74.1 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.31.48.193 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 46.137.171.215 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a05:d018:76c:b685::de70 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a05:d018:76c:b680::de70 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a05:d018:76c:b684::de70 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.43.88.59 for host meechum.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 50.112.200.172 for host meechum.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 34.209.107.23 for host meechum.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.127.185.23 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 40.127.185.23 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 65.52.144.94 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 65.52.144.94 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.113.92.244 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 40.113.92.244 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.113.83.221 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 40.113.83.221 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.46.33.89 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 104.46.33.89 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 65.52.151.72 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 65.52.151.72 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 137.116.199.193 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 137.116.199.193 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.40.156.160 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 104.40.156.160 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.115.58.225 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 23.97.209.231 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 23.97.169.223 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.113.92.113 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.127.133.56 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 23.102.24.87 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.127.179.193 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.115.10.186 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: SKYPE_HOSTS hostname: dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 104.40.156.160
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 137.116.199.193
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 65.52.151.72
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 104.46.33.89
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.113.83.221
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.113.92.244
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 65.52.144.94
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.127.185.23
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.115.58.225 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 23.97.209.231 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 23.97.169.223 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.113.92.113 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.127.133.56 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 23.102.24.87 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.127.179.193 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.115.10.186 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table SKYPE_HOSTS host: dr.skype.net error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]: failed to resolve host directory.services.live.com will retry later again.
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.76.0 for host peoplegraph.firstpartyapps.oaspapps.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 35.214.201.112 for host www.eztitles.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.213.44 for host signup.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.246.44 for host signup.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:1ec:bdf::44 for host signup.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:1ec:46::44 for host signup.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.69.106.216 for host dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 13.69.106.216 for host dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.236.186.217 from host dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: dc.services.visualstudio.com address: 13.69.106.216
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: dc.services.visualstudio.com address: 52.236.186.217 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: dc.services.visualstudio.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.246.44 for host mem.gfx.ms
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.213.44 for host mem.gfx.ms
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:1ec:bdf::44 for host mem.gfx.ms
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:1ec:46::44 for host mem.gfx.ms
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.79.197.34 for host pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 40.79.197.34 for host pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.168.117.169 from host pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.113.130.243 for host myanalytics.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: SKYPE_HOSTS hostname: pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: pipe.skype.com address: 40.79.197.34
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: pipe.skype.com address: 52.168.117.169 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table SKYPE_HOSTS host: pipe.skype.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.113.199.175 for host api.asm.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 23.197.104.24 for host omextemplates.content.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.76.42.76 for host myanalytics-gcc.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.174.193.75 for host get.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 20.44.10.123 for host mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 20.44.10.123 for host mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.79.197.35 from host mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: SKYPE_HOSTS hostname: mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: mobile.pipe.aria.microsoft.com address: 20.44.10.123
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: mobile.pipe.aria.microsoft.com address: 40.79.197.35 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table SKYPE_HOSTS host: mobile.pipe.aria.microsoft.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.6.162 for host officespeech.platform.bing.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.76.22 for host testconnectivity.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.111.243.9 for host augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.111.243.9 for host augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.111.243.8 from host augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: augloop.office.com address: 52.111.243.9
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: augloop.office.com address: 52.111.243.8 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: augloop.office.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.69.106.89 for host dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 13.69.106.89 for host dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.236.186.217 from host dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: dc.applicationinsights.microsoft.com address: 13.69.106.89
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: dc.applicationinsights.microsoft.com address: 52.236.186.217 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: dc.applicationinsights.microsoft.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.20.9 for host office.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2603:1030:800:5::bfee:ac7d for host office.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 51.116.60.32 for host management.azure.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www5.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www5.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www3.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www3.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www7.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www7.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www4.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www4.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www6.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www6.atomicorp.com

I thought this would have been fixed with 2.6. I had to reenable the cron workaround. Oh well.

Can you guys try out below workaround for max threads per process? I have been suffering now for weeks with this issue, rendering my firewall unreliable and useless, issues with VoIP etc. I also reverted back from 2.6 to 2.5.2, no change. But it seems, that for me this workaround solved the issue- it remains open if that is just temporary... I will now try upgrading back to 2.6.0 and retest.

[[https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html]]
Let me know your results. This issue is very annoying.

As a Sidenote: after updating to 2.6.0 a once working ruleset completely broke. I have now restored the backup and again works fine on 2.5.2. What a mess.

Been fighting this issue on 2.5 and 2.4.5 and I am talking about using only 1 DNS entry in the Alias to a Dynamic DNS on several firewalls - intermittent with various ones and difficult to pin down. Will try a different firewall at one of the site and see if that occurs; another 'sense.

Same here, plase fix it.

I added the setting from https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html and will give it some time to see if this fixes the issue.

It seems this issue has gotten worse somewhere along the line similar to how others are describing it. Tables now load empty rather than only partially loading.

Also there is a related bug likely tied to the same sanity checks (or lack of):
https://redmine.pfsense.org/issues/13282

I'm having a crack at this issue now. Is everyone experiencing this issue using unbound as a resolver by chance?

Reid Linnemann wrote in #note-101:

I'm having a crack at this issue now. Is everyone experiencing this issue using unbound as a resolver by chance?

Since unbound is the default, I'd imagine it's related. I have only seen this crop up with unbound being used as DNS for the firewall first, with remote second. Will need to confirm next time I can reproduce it.

There are several things I've noted about how aliases and filterdns work that - if they aren't directly related to this problem - certainly don't help. I'm going to start by going down those rabbit holes and see where they take me.

I was suspecting unbound caching behavior could get in the way if filterdns was issuing ANY queries, but I have since ruled that out - filterdns uses getaddrinfo, which makes A and AAAA queries.

Had to create an account just to leave a note regarding this issue

We host a text file at https://www.mydomainurl.com/list/ip.txt (not the real domain)
This is our "traveling webgui access list" - its literally a flat txt file with 10-15 ip addresses listed one per line
We have the URL setup in pfsense as an Alias URL Table
We have cron set to update urltables every 2mins with "now forceupdate"
When traveling or at a remote site and needing temp access to a router, we will update this txt list via ftp and cron will pull a new copy
When the new copy is pulled, the updated IP we added to the list is now part of our "webgui access rule"
Now we can login to the router from that new IP we added, and when done we'll remove the IP from the list and upload a new copy via ftp
Log file shows every 2mins URL table updates with either "no changes" or "1 address added/removed"

Earlier today, for some reason the firewall stopped updating this alias URL table???
The list that shows is "stuck" and has our 9 IP addresses listed
Attempts to delete and recreate it yield the EXACT SAME 9 IP addresses again with NO changes or updates.
Attempts to add a new list, named differently, which is added as a new alias and new rule, which is even hosted at a different URL (like github) yield the EXACT SAME ORIGINAL 9 IP addresses!!?!
Mousing over the existing alias rule and the new alias rule - the same 9 IP addresses are listed, but the URLs shown at the top of each list are different. Very weird.
If you browse up each URL in the web browser, they show different correct new lists (more than 9 IPS listed)

• Rebooting the router does not help
• Updating the tunables with the kern.threads.max_threads_per_proc = 4096 does not help (even post reboot)
• pkill filterdns and reloading it does not help

And here's the kicker: we have approx 70 pfsense boxes we run, all on 2.6 or 22.01 (either virtual or purchased from Netgate if a physical box), and all 70 have the original list stuck at the same point with the same 9 IP addresses from the last time they all updated. I've logged into most all of them the past 4hrs and can confirm they all show the list stuck at the same point despite the logs showing cron running, confirming it checks the URL and confirms there are "no changes" despite me updating the URL list via ftp to 10+ IP addresses.

I have tested updating the aliases, URLs, creating new ones, new rules etc as described above on at least 4 of them so far and they all exhibit the same behavior.

Creating standard aliases with individual host IP entries works fine, which is how I've had to set all of them currently.

We have had this setup for the past 3+ yrs with ZERO issues and pfsense for us since 2012 has been extremely dependable and stable, but today, not sure what the heck happened? Seems like this specific issue is ongoing and now bleeding into 2.6 and 22.01... Also wanted to make it known this has a very real impact regarding sense of stability for us.

Open to suggestions on how to fix.

UPDATE: Tinkering some more this morning. Found out that if I make a new alias URL table, point it to a new URL list that starts with an invalid IP like 10.2.3.4, let it update that IP once, then I can add subsequent IPs to the list and the list now updates normally... now in the process of updating each firewall with the new alias/list and removing the old alias/list.

Bob, thank you for your detailed report. Can you confirm for me that all of the entries in the hosted list are IPs, and no FQDNs are used?

Some of the issues with FQDNs are better with 2.6/2.7.0-development and 22.05, but there are still very real problems that seem to stem from filterdns. I have a reliable way to generate incomplete tables at this point that may shed some light on what's going on.

@Reid - per your previous question - yes our entire list is only IP addresses with a #comment after each address. No FQDNs in the list at all. The alias itself is a URL table with our FQDN: https://www.mydomainurl.com/list/ip.txt (not the real domain)

example of what the list is
0.0.0.0 #comment
1.2.3.4 #comment
etc.
etc.

I think I have this pretty well nailed down to a race on initial thread creation for added aliases where the created thread is not waiting on a condvar before the routine that spawned threads cycles through them and signals them, which leads to the initial run through the action routine that adds the address to the alias table being delayed by the filterdns interval (5 minutes). I believe I will have a fix for snaps soon.

I have the same issue. Mixing FQDN and IP addresses caused me hours of frustration why various rules were not working. Finally seeing this bug report and the reported work-around of changing the timeout for Alias resolution worked.

I am using version:
2.6.0-RELEASE (amd64)
built on Mon Jan 31 19:57:53 UTC 2022
FreeBSD 12.3-STABLE

Brian M wrote in #note-116:

I have the same issue. Mixing FQDN and IP addresses caused me hours of frustration why various rules were not working. Finally seeing this bug report and the reported work-around of changing the timeout for Alias resolution worked.

That's a really hit-and-miss workaround, I fixed some very troublesome thread synchronization issues in filterdns that were the root cause. If you can update to 2.7.0-devel I'd really recommend it, especially if you're making extensive use of FQDN aliases.

I am testing https://snapshots.netgate.com/amd64/pfSense_master/installer/pfSense-CE-2.7.0-DEVELOPMENT-amd64-20221216-0600.iso.gz

There are same problems as in v2.4.5p1.

1) Add following domains to alias (conventional names and ip's):
aa.dyndns.org (2.2.2.2)
bb.dyndns.org (3.3.3.3)
cc.dyndns.org (4.4.4.4)
dd.dyndns.org (5.5.5.5)

2) Table will be correctly populated.

3) Change aa.dyndns.org to 3.3.3.3 in dyndns (duplicate of bb.dyndns.org), wait.

4) Table will become correct (3 entries, duplicates removed).

5) Change aa.dyndns.org back to 2.2.2.2 in dyndns, wait.

6) Table will not be correct anymore. It will miss 3.3.3.3 entry no matter how long you wait.

7) Clear table using red "Empty table" button in Diagnostic->Tables

8) Table will remain clear and won't be filled anymore, until restart.

It looks like incorrect caching of table entries.

Adding new (ADDR_NEW) and removing old (ADDR_OLD) address from pf table on dns update would work if duplicates were not removed from table, and won't work if duplicated addresses exist in different dns records, and then removed in pf table.

Allowing user to perform "Empty table" which brings tables to inconsistent state until firewall restart is also a strange solution.

And I have to agree with others - the lack of proper testing of such basic functionality, and not fixing it for years is not about "world's most trusted open source network security solution"...

Unfortunately, the exact thing happened again in 2.7.0 for us over the weekend. We use an external spamfilter where mail from our customers flows through. We have an alias for their DNS records, and some of them are in by IP as well (as they might change their DNS records over time). Last saturday mail stopped coming in. And looking at the tables, we had the same issue again; one IP was in as IP directly, and as FQDN. And the resulting IP was just not in the table for the alias.

Now this is only the first time we've had this happen in 2.7.0, but so far I consider this issue still not fixed. Are there any logs I can / should paste?

[edit] In the meanwhile I got closer to the issue; in the weekend our spamfilter changed one DNS record to an IP we had in that same alias statically. So, we had x.x.x.x statically, and some.dns.entry which now resolves to x.x.x.x as well. Result: duplicate entry, which was removed in the table.

This is an issue that's described extensively in this bugtracker thread, and as per 2.7.0 I still consider it not solved.

[edit2]
it's reproducible as well:

- Create an alias
- add 1.1.1.1
- add 8.8.8.8
- add a dns entry you created, pointing to 1.1.1.1, ie pfsensetest.domain.com
- monitor the table-entry for the alias, all will be ok
- now change the DNS entry from 1.1.1.1 to 8.8.8.8 and wait for it to be replicated
- in my setup, 1.1.1.1 got deleted from the table. So while 8.8.8.8 is in there 'twice' now, and 1.1.1.1 only once, it's not there anymore.
- killing filterdns and reloading filters repopulates the tables correctly it seems.

Tried 3 times, all with the same results.