Project

General

Profile

Actions

Bug #9296

closed

Alias content is sometimes incomplete when an alias contains both FQDN and IP address entries

Added by Ph. T about 5 years ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Category:
Aliases / Tables
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
23.01
Release Notes:
Affected Version:
2.5.0
Affected Architecture:
All

Description

If you are using FQDN-Aliases each FQDN can only be used once, if
you use the alias twice, the generated tables are incomplete.

No DNS-Server/Resolver on the firewall is used. External DNS
resolvers are configured.

Example:
alias1 : fqdn1, fqdn2, fqdn3
alias2 : fqdn4, fqdn2, fqdn3

Generated tables are incomplete
alias1 : fqdn1, fqdn2, fqdn3
alias2 : fqdn4 (the others are missing)

alias2 does only contain fqdn4 and fqdn2 and fqdn2 are missing.

This bug seems to arise with 2.4.4_p1 and is still existing in 2.4.4_p2;
I am not sure if this behavior is present within 2.4.4.

I am working on a minimal example which i will provide.


Files

Rule_Set.PNG (37.4 KB) Rule_Set.PNG Ruleset Ph. T, 01/31/2019 05:29 AM
Alias_Configuration.PNG (24.2 KB) Alias_Configuration.PNG Alias-Configuration Ph. T, 01/31/2019 05:29 AM
table_fqdn1.PNG (35.8 KB) table_fqdn1.PNG table fqdn1 Ph. T, 01/31/2019 05:29 AM
table_fqdn2.PNG (49.6 KB) table_fqdn2.PNG table fqdn2 Ph. T, 01/31/2019 05:29 AM
191011_Tnk_config-pfSense.localdomain-20191011143458.xml (15.9 KB) 191011_Tnk_config-pfSense.localdomain-20191011143458.xml Ph. T, 10/11/2019 09:40 AM
pfsense.png (39 KB) pfsense.png Art Manion, 10/31/2019 11:48 PM
filterdns-2.0_3.txz (17.5 KB) filterdns-2.0_3.txz filterdns pkg built by me on a FreeBSD 11.2 VM Eduard Rozenberg, 02/01/2020 10:19 AM
clipboard-202202191221-tmdxs.png (15.7 KB) clipboard-202202191221-tmdxs.png → luckman212, 02/19/2022 11:21 AM
filterdns_verbosity.diff (2.73 KB) filterdns_verbosity.diff Marcos M, 05/24/2022 02:32 PM
clipboard-202208041447-c1pbe.png (10.2 KB) clipboard-202208041447-c1pbe.png File Compare Marco Jäger, 08/04/2022 07:47 AM
clipboard-202208041453-rcv9i.png (86.8 KB) clipboard-202208041453-rcv9i.png Broken Alias Marco Jäger, 08/04/2022 07:53 AM
clipboard-202208041455-mnwun.png (175 KB) clipboard-202208041455-mnwun.png Working Alias Marco Jäger, 08/04/2022 07:55 AM
filterdns (35.9 KB) filterdns New Filterdns file from pfsense version 2.6.0 Marco Jäger, 08/04/2022 08:00 AM
test2_alias.png (73 KB) test2_alias.png Azamat Khakimyanov, 10/10/2022 06:07 AM

Related issues

Related to Bug #12708: Alias with non-resolving FQDN entry breaks underlying PF tableResolvedReid Linnemann

Actions
Related to Bug #13067: Resolve interval for ``filterdns`` may not match the configured valueResolvedReid Linnemann

Actions
Related to Bug #13792: Filterdns assumes sets of resolved addresses for each hostname are nonintersectingNewReid Linnemann

Actions
Related to Bug #13793: filterdns does not reconcile modelled tables with the current state of filter tablesNewReid Linnemann

Actions
Actions #1

Updated by Ph. T about 5 years ago

I have now prepared a minimal example:

As you can see fqdn1 is missing the entry for one.one.one.one

Please FIX

Actions #2

Updated by Eduard Rozenberg about 5 years ago

I believe my issues may be related to this. We updated to 2.4.4 p2 on Jan 9, but only in the past few days have seen the problems.

The firewalls and sites at several locations are configured to allow remote access based on firewall aliases and rules using those aliases. The aliases sometimes contain a mix of IP addresses (/32), IP networks (/29 for ex), and DNS names (something.company.com).

Since the past few days, the pfSense firewalls at the various sites all reject my remote connection attempts, and the rejections are visible in the firewall logs.

It's...a problem.

Actions #3

Updated by Jim Pingle about 5 years ago

  • Category set to Rules / NAT
  • Assignee set to Luiz Souza
  • Target version set to 48
  • Affected Architecture All added
  • Affected Architecture deleted ()
Actions #4

Updated by Robert Gijsen about 5 years ago

2.4.4-RELEASE-p2, I've had this multiple times. At the moment I can even sort of reproduce it.
When adding hosts to an alias my AD DNS server logs:

2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Rcv <pfsense IP> a463 Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Rcv <pfsense IP> 519a Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Snd <pfsense IP> a463 R Q [8081 DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Snd <pfsense IP> 519a R Q [8085 A DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

This is an external host, i.e. a DNS that needs to be externaly resolved by our DNS servers. That seems to work fine result gets send back to pfSense. However the host does NOT end up in the table for that alias. When I add another DNS, same domain, so hosted at the same DNS on internet, that works fine. I tried others like www.tweakers.net, www.nos.nl or bbc.co.uk I have the same success loggings in my DNS debug log, and they DO end up in the alias table as well.

pfSense Resolver log:
Feb 18 12:47:14 filterdns Adding host <Host that gets added to the alias> (I just added that one to the alias)
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <Host that gets added to the alias>
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <host that does NOT end up in table> (I just added that one in the alias as well)
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <existing host, which was already in the alias>

The host that does NOT end up in table here, is by the way successfully added to some other aliasses, where it works just as expected. But for this alias I am missing the 'Adding host' in the pfSense log.

I tried creating a new alias, with the same three hosts as in the alias I used above. Here NONE of them end up in the table, after waiting for about 20 minutes, while in the alias used above two out of three (and the same two every time, no matter what order I put them in) work. Then I added www.tweakers.net as another try, and that one gets in there immediately.
I again killed filterdns, restarted it and poof - the tables immediately got filled as they should. So it seems filterdns is partially functional - some hosts get added, some aren't. It could indeed be when hosts already exist in the table somewhere; however restarting filterdns at least populates them for a while.

Tell me what loggings you need. As it seems I can now reproduce this at will (also on my second carp / HA node by the way) I can probably give all needed logs.

Actions #5

Updated by Robert Gijsen about 5 years ago

I've just downgraded a test-machine to 2.4.4 release, and that works fine. Keeping it there for a while.

Actions #6

Updated by Eduard Rozenberg about 5 years ago

Shortly after I posted my problem above 20 days ago, it started working again on its own.

Then today, it is again not working.

So it may be a sporadic issue with the alias resolution, that doesn't happen consistently. Have not been able to pin down the issue at all.

Actions #7

Updated by Eduard Rozenberg about 5 years ago

I can confirm my issue is the same as described by the other posters on this bug.

Logs show that filterdns claims to be doing the right thing - all expected alias entries (FQDN's, IP's, networks) show up in:
$ clog /var/log/resolver.log | grep "Adding Action"

But the alias table is incomplete, some IP addresses are missing:
$ pfctl -T show -t my_alias_name

There is no DNS resolution issue with any of the FQDN's - if I ping the FQDN's from the firewall their IP addresses are resolved.

Restarting the filter, re-saving the alias does not help.

Actions #8

Updated by Eduard Rozenberg about 5 years ago

I've also ruled out some other possibilities below -

Not the issue:
https://docs.netgate.com/pfsense/en/latest/firewall/thread-error-using-many-hostname-in-aliases.html
(I don't have a threads error in logs, and setting this tunable did not help)

Not the issue:
Mixing FQDN's and IP's - I tried creating a new alias with only a single FQDN from the ones that don't work in the original alias. Still no luck.

Actions #9

Updated by Jim Pingle about 5 years ago

  • Target version changed from 48 to 2.5.0
Actions #10

Updated by Azamat Khakimyanov almost 5 years ago

I see this behavior on 2.4.4_p2, on 2.4.5-dev and on 2.5.0-dev.
As workaround we can:
- in console run 'pkill filterdns' command
- then /Status/Filter Reload to start 'filterdns' service

Actions #11

Updated by Gavin Stewart almost 5 years ago

As a workaround I have installed the Cron package with the following additional entries:

*/15 * * * * root killall -9 filterdns; sleep 2; /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1     
@reboot      root sleep 10; killall -9 filterdns; sleep 2; /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1

Actions #12

Updated by Robert Gijsen almost 5 years ago

I know it's targeted for 2.5.0, but still I'd like to inform people here that 2.4.4_3 does indeed NOT fix this, making it yet another update that kills main functionality. Be aware, and be extremely reluctant to update!

Actions #13

Updated by Rudolf Mayerhofer almost 5 years ago

Setting "Aliases Hostnames Resolve Interval" to 30 seconds (which should be the minimum value) in System/Advanced/Firewall&NAT seems to work around the issue for me (which could be some kind of race condition in filterdns but that's just guessing on my side).

Actions #14

Updated by Chris Tsou almost 5 years ago

Rudolf Mayerhofer wrote:

Setting "Aliases Hostnames Resolve Interval" to 30 seconds (which should be the minimum value) in System/Advanced/Firewall&NAT seems to work around the issue for me (which could be some kind of race condition in filterdns but that's just guessing on my side).

Based on what Rudolf wrote, I changed the value of "Aliases Hostnames Resolve Interval" from empty to 300secs. That setting combined with a filter reload from "Status/Filter Reload" menu made my rules work as expected again.

I will continue testing and update this post if I find anything else.

Actions #15

Updated by Gavin Stewart over 4 years ago

Christoforos Tsoukaris wrote:

Based on what Rudolf wrote, I changed the value of "Aliases Hostnames Resolve Interval" from empty to 300secs. That setting combined with a filter reload from "Status/Filter Reload" menu made my rules work as expected again.

I will continue testing and update this post if I find anything else.

If you look at the cron entries I have mentioned earlier (#9296#note-11), you will see that I have the interval (-i) set to 300. I am still seeing missing entries in the alias tables on occasion, which do get corrected when cron kills and restarts filterdns within the next 15 mins.

Actions #16

Updated by Mark Monaghan over 4 years ago

The crontab entries as mentioned in #11 didn't run as they were just keeping on adding new filterdns processes, eventually causing the firewall to trigger CARP/HA, give high latency to VPN and internet traffic, and eventually cause the firewall to stop passing traffic altogether. I went for installing the cron GUI package (But you could just as easily edit /etc/crontab directly), and I've changed the lines to:

*/15 * * * *  root    /usr/bin/pkill -f "/usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1"; sleep 2; /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1
@reboot       root    sleep 10; /usr/bin/pkill -f "/usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1"; sleep 2; /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1

I've checked that these cron jobs now correctly kill the old processes before starting new ones, and that the filterdns process is correctly doing it's job as long as the restarts are in place (As noted in other notes to this bug, it still stops resolving after a while, although I've not had time to monitor the firewalls to find out exactly how long it is before the process stops, so apologies for that).

Actions #17

Updated by Gavin Stewart over 4 years ago

Mark Monaghan wrote:

The crontab entries as mentioned in #11 didn't run as they were just keeping on adding new filterdns processes,

Very interesting. I'm not seeing that occur at all (and it was something I was monitoring closely when I set it up). I wonder what makes this operation different on your system ? Could you have possibly forgotten the "-9" argument to "killall" ?

Actions #18

Updated by Mark Monaghan over 4 years ago

Gavin Stewart wrote:

Very interesting. I'm not seeing that occur at all (and it was something I was monitoring closely when I set it up). I wonder what makes this operation different on your system ? Could you have possibly forgotten the "-9" argument to "killall" ?

No, sorry, I copied and pasted the commands verbatim from here to ensure that I didn't make any errors when implementing them. The -9 was definitely in there. What I was finding was that if I ran them individually, or even as a grouped command set, from the CLI, they worked perfectly, but they failed to kill any processes when ran via the cron. This was all done and tested on 2.4.4-p3. I cannot comment on how this performed prior to this version, as it wasn't implemented on 2.4.4-p2 or lower, only after the system was upgraded to run on the latest stable version.

This is the reason I switched the cron job to pkill as nothing I tried would get killall or even kill to terminate the filterdns process via the cron, but pkill was working for reasons only known to the OS. However, that also presented it's own challenges as unless I used the exact filter of -f "/usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1" rather than pkill filterdns or pkill -f "filterdns" it was seeing filterdns in it's own cron job and killing itself before it killed any filterdns processes, so until I put the large filter in place, I wasn't any further forward and still had filterdns processes stacking up, causing the system to fall over eventually (I put this down to the system running out of open file handles as it certainly never ran out of memory before the crashes that I experienced started to happen.

Actions #19

Updated by Rudolf Mayerhofer over 4 years ago

Rudolf Mayerhofer wrote:

Setting "Aliases Hostnames Resolve Interval" to 30 seconds (which should be the minimum value) in System/Advanced/Firewall&NAT seems to work around the issue for me (which could be some kind of race condition in filterdns but that's just guessing on my side).

As a follow up: With 30 seconds resolve interval things are still working fine one month later without killing/restarting filterdns.

Actions #20

Updated by Eduard Rozenberg over 4 years ago

Rudolf Mayerhofer wrote:

As a follow up: With 30 seconds resolve interval things are still working fine one month later without killing/restarting filterdns.

Unfortunately this doesn't solve my problem. Same issues, regardless of the refresh value. Continues to make life difficult on a regular basis.

Actions #21

Updated by Peter van der Kleij over 4 years ago

I think I have a similar problem.
My inbound rule did not work with an FQDN in the Alias. (Whitelist for source addresses) Weird thing was that only one ip-address in the Alias (not the FQDN) did not work, restarting servers/pfsense and such did not give any result.

When I enable the 300 sec for 'Aliases Hostname Resolve interval', it WORKS, when i leave it empty, it FAILS directly.

2.4.4-RELEASE-p3, FreeBSD 11.2-RELEASE-p10

Actions #22

Updated by Art Manion over 4 years ago

Netgate SG-4860 running 2.4.4-RELEASE-p3 (amd64). At least twice I've experienced issues, I assume involving filterdns, where aliases were updated via the GUI but did not make it into the pf table (and I believe the updates didn't land in /var/etc/dnsfilter.conf either, but I can't confirm).

The alias/table is 58 entries, mostly DNS names, some IP addresses, and some other pfSense aliases. The last time this problem happened neither DNS name, IP address, nor pfSense aliases were added to the pf table. filterdns was running. killall -9 filterdns && /etc/rc.filter_configure fixed it.

Actions #23

Updated by Justin J over 4 years ago

Also experiencing this issue on 2.4.4-p2 and now 2.4.4-p3. If FQDNs are remove the table updates correctly. Due to the way many cloud service provider clusters move and reallocate IPs it is impractical to use individual host or CIDR lists as this requires constant updating once a change is made on the remote server. In my case I first noted the issue after one entry was updated and became a duplicate IP in a CIDR that was in the table. I've tried killing and restarting filterdns and changing resolution time to 30 seconds but with no improvement. Sometimes the table would partially generate, other times it was completely empty.
Can this be escalated for a resolution before 2.5 as it breaks the core firewall functionality for those of us using a cloud or hybrid cloud setup?

Actions #24

Updated by Robert Gijsen over 4 years ago

I second Justins message / question. pfSense is completely unusable after 2.4.4 initial release. With filterdns not working properly, it fails as a firewall completely. That leaves us with the choice of updating pfSense to a completely useless not working version, or leave it at 2.4.4 and leave it vulnerable to now known security issues. Both are unacceptable.

This should certainly get higher priority.

Actions #25

Updated by Tom Hebert over 4 years ago

Most of you are more experienced at this than me so please be tolerant if this is a dumb question.

I added a Firewall Alias containing a single FQDN, xxx.mydomain.com. A Diagnostics=>DNS Lookup for xxx.mydomain.com returns three addresses, one IPv4 and two IPv6 (received from DHCPv6 and RA). However when I inspect the alias's table using Diagnostics=>Tables, only the IPv4 address is listed. It follows that the rule referencing the alias isn't passing traffic to the IPv6 addresses. I would expect to see all three addresses in the table. Is that expectation correct or am I missing something?

For background, I am using resolver and there is a domain override in place for mydomain.com (TypeTransparent).

Am I experiencing this bug?

Actions #26

Updated by Justin J over 4 years ago

That sounds like it might be something else Tom. Check your output from the CLI with: pfctl -T show -t ALIASNAME
If it's not there try making a forum post as the discussion here should be directly about the bug, not diagnosing the possibility of your setup having it.

Actions #27

Updated by Tom Hebert over 4 years ago

Justin J: I took your advice and posted on the forum and was promptly referred back here. Here's the link in case you are inclined to read it. https://forum.netgate.com/topic/145553/firewall-alias-not-updating-table-correctly

Actions #28

Updated by Jim Pingle over 4 years ago

  • Category changed from Rules / NAT to Aliases / Tables
Actions #29

Updated by Robert Gijsen over 4 years ago

It's been about 8 months now that we are unable to update / patch our firewalls because of this. Yeah I know, open source, contribute yourself if you don't like it, and so on. But still no update on this from pfSense team? By now we are seriously considering moving away. It's unacceptable that we have to run a firewall that's by now simply not secure anymore because of the now public security flaws.

Can we get an official status update on this? And why this doesn't get higher priority? Is there an official ETA for 2.5?

Actions #30

Updated by John K over 4 years ago

Robert Gijsen wrote:

It's been about 8 months now that we are unable to update / patch our firewalls because of this. [...] And why this doesn't get higher priority? Is there an official ETA for 2.5?

This issue is becoming a show stopper for us as well.

Actions #31

Updated by Angel Briceño over 4 years ago

Ph. T wrote:

If you are using FQDN-Aliases each FQDN can only be used once, if
you use the alias twice, the generated tables are incomplete.

No DNS-Server/Resolver on the firewall is used. External DNS
resolvers are configured.

Example:
alias1 : fqdn1, fqdn2, fqdn3
alias2 : fqdn4, fqdn2, fqdn3

Generated tables are incomplete
alias1 : fqdn1, fqdn2, fqdn3
alias2 : fqdn4 (the others are missing)

alias2 does only contain fqdn4 and fqdn2 and fqdn2 are missing.

This bug seems to arise with 2.4.4_p1 and is still existing in 2.4.4_p2;
I am not sure if this behavior is present within 2.4.4.

I am working on a minimal example which i will provide.

I had the same problem. The rules stack has a limit and this means that domain names cannot be resolved. For example:

Alias-1 => Network => "10.10.0.0/24" It is not equal to "10.10.0.0-10.10.0.254"

While 10.10.0.0/24 is a valid nomenclature for a rule, but 10.10.0.0-10.10.0.254 say that the entire range should be described:
10.10.0.0
10.10.0.1
10.10.0.2
10.10.0.3
10.10.0.4
..
...
....
10.10.0.254

This range of IPs causes a problem for the "next" aliases in the system, and it is very possible that they cannot be resolved.

I have removed all gigantic ranges of IPs and the problem is solved.

This problem has already affected several cloud providers, so they mostly do not accept using FQDN aliases in their routing rules or ACLs.

Actions #32

Updated by Gavin Stewart over 4 years ago

Angel Briceño wrote:

I have removed all gigantic ranges of IPs and the problem is solved.

I have no ranges of IP addresses (only networks defined in CIDR notation), and the problem persists.

Actions #33

Updated by Ph. T over 4 years ago

I am very,very unhappy with the time it takes to deal and fix this problem.
Is there any way to speed up the process ? I can provide any additional info.

I think Angel has a complete different problem. The limit of table entries
has been an issue due to big bogon tables some time ago. A fix might be to
increase the maximum table entries, or not to use the bogon table.

    System > Advanced >Firewall & NAT
    Firewall Maximum Table Entries 

we have set this value to 400000.

Actions #34

Updated by Luiz Souza over 4 years ago

Ph. T wrote:

I am very,very unhappy with the time it takes to deal and fix this problem.
Is there any way to speed up the process ? I can provide any additional info.

I think Angel has a complete different problem. The limit of table entries
has been an issue due to big bogon tables some time ago. A fix might be to
increase the maximum table entries, or not to use the bogon table.
[...]
we have set this value to 400000.

Please, provide the filterdns logs for your case, with debug enabled (-d20).

As strange as it may seem, this is proving to be difficult to reproduce reliably.

If you want to send the logs privately, please send it to luiz at netgate.com

Thanks.

Actions #35

Updated by Jim Pingle over 4 years ago

If anyone can come up with simple cases that reliably reproduce the problem, that would definitely help. That is, the smallest possible configuration that results in the problem happening. For example, an alias and firewall rule which exhibit the problem (or multiple aliases+rules) along with whatever other conditions are necessary, such as waiting specific amounts of time, or having invalid hostnames in the alias, etc. Along with log data mentioned above and the contents of /var/etc/filterdns.conf.

Actions #36

Updated by Ph. T over 4 years ago

I will provide the data / config.xml . I could also provide a virtual-box pfsense-installation
which shows this problem. I hope i could provide this today.

Actions #37

Updated by Ph. T over 4 years ago

I have tried to reproduce the issue. Unfortently that was not possible. Now i just get complete empty tables.
I have waited the timeout.

I have used a 2.4.4_p3 Image as base. I turned of DNS forwarder and Resolver;Using an external resolver.

Steps to reproduce:

Start the machine:

Delete entry using_one_alias
Delete entry using_one_alias2

Add entry
using_one_alias host, fqdn_alias2
using_one_alias2 host, fqdn_alias1

Without killing the filterdns-process thouse tables remain empty.

[2.4.4-RELEASE][]/var/log: cat /var/etc/filterdns.conf
pf dns.google fqdn_alias1
pf one.one.one.one fqdn_alias1
pf one.one.one.one fqdn_alias2
pf one.one.one.one using_one_alias
pf dns.google using_one_alias2
pf one.one.one.one using_one_alias2

If you do a reboot the tables are populated.

But if you delete the aliases after a reboot and recreate them the tables are not filled until you restart filterdns.

Actions #38

Updated by Ph. T over 4 years ago

I see similar effects with the old config which i attached in January.

Actions #39

Updated by John K over 4 years ago

Jim Pingle wrote:

If anyone can come up with simple cases that reliably reproduce the problem [...]

What's the status here? Has Netgate been able to reproduce this issue?

Actions #40

Updated by Jim Pingle over 4 years ago

John K wrote:

What's the status here? Has Netgate been able to reproduce this issue?

Not that I have seen yet. We still need to find a combination of settings that reliably and repeatedly reproduces the issue.

Actions #41

Updated by Vinicius DellAglio over 4 years ago

Jim Pingle wrote:

John K wrote:

What's the status here? Has Netgate been able to reproduce this issue?

Not that I have seen yet. We still need to find a combination of settings that reliably and repeatedly reproduces the issue.

I just installed a brand new pfsense box and once I created an alias with an FQDN it didn't work, when I checked it on TABLES I only had the entries' list above the fqdn entry, the fqdn itself and everything else were missing.

Once I deleted the FQDN entry and hit filter reload, the alias was correct.

Actions #42

Updated by John K over 4 years ago

Vinicius DellAglio wrote:

I just installed a brand new pfsense box and once I created an alias with an FQDN it didn't work, when I checked it on TABLES I only had the entries' list above the fqdn entry, the fqdn itself and everything else were missing.

Once I deleted the FQDN entry and hit filter reload, the alias was correct.

Is the FQDN a CNAME, A record, etc?

Actions #43

Updated by Art Manion over 4 years ago

Art Manion wrote:

Netgate SG-4860 running 2.4.4-RELEASE-p3 (amd64). At least twice I've experienced issues, I assume involving filterdns, where aliases were updated via the GUI but did not make it into the pf table (and I believe the updates didn't land in /var/etc/dnsfilter.conf either, but I can't confirm).

The alias/table is 58 entries, mostly DNS names, some IP addresses, and some other pfSense aliases. The last time this problem happened neither DNS name, IP address, nor pfSense aliases were added to the pf table. filterdns was running. killall -9 filterdns && /etc/rc.filter_configure fixed it.

Update:

In the GUI, there is an alias named yoke11 that contains the IP address 192.168.1.211. There is another alias, internet_allowed_LAN, that contains yoke11. Weeks after this configuration was made (including multiple save/apply actions from the GUI), 192.168.1.211 is not the internet_allowed_LAN pf table and is in /var/etc/filterdns.conf. This vaguely points to a problem between filterdns.conf and the actual pf table.

I tried once to reproduce this behavior with clean/new/test aliases and I did not observe the problem.

pfctl -t internet_allowed_LAN -T test 192.168.1.211
0/1 addresses match.

grep '192.168.1.211' /var/etc/filterdns.conf
pf 192.168.1.211 internet_allowed_LAN

killall -9 filterdns
/etc/rc.filter_configure

pfctl -t internet_allowed_LAN -T test 192.168.1.211
1/1 addresses match.

I also observe that aliases do not become pf tables until the alias is used in a firewall rule. I believe this is expected behavior, just noting it.

Actions #44

Updated by Gavin Stewart over 4 years ago

Jim Pingle wrote:

John K wrote:

What's the status here? Has Netgate been able to reproduce this issue?

Not that I have seen yet. We still need to find a combination of settings that reliably and repeatedly reproduces the issue.

I now have a minimal and repeatable set of steps to reproduce this.

This has been verified in a VirtualBox VM with the following configuration:
  • New VM for FreeBSD 64-bit, accepting all defaults
  • First network adapter stays NAT (used as WAN in pfSense)
  • Add second network adapter as host-only (used as LAN in pfSense) to access web interface from host
  • pfSense-CE-2.4.4-RELEASE-p3-amd64.iso.gz, accepting all defaults
  • configure LAN IP address as needed at pfSense console to match host-only network settings
Procedure to reproduce FQDN name resolution failure in Alias tables:
  1. Firewall -> Aliases -> Import
  2. Alias Name "TEST1", import entire alias list below, Save
  3. Diagnostics -> Tables -> "TEST1"
    + Note approx 59 entries, specifically the 192.168.x.x addresses.
    + (It may take a couple of page reloads for all the addresses to resolve.)
  4. Firewall -> Aliases -> Import
  5. Alias Name "TEST2", import entire alias list below, Save
  6. Diagnostics -> Tables -> "TEST2"
    + Note that table is never fully populated (even if waiting longer than the 5 minute filterdns interval).
    + Note that 192.168.x.x addresses do not all appear.
  7. Kill the filterfdns process at the console, and restart manually:
    /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
    + Note that "TEST2" table is now properly populated.

List of aliases to import:

mail.bigpond.com
speedtest.telstra.com
mirror.internode.on.net
192.168.1.1
mirror.aarnet.edu.au
account.mojang.com
authserver.mojang.com
192.168.2.2
mc.hypixel.net
sessionserver.mojang.com
launchermeta.mojang.com
192.168.3.3
apps.yourtown.com.au
obook4.oxforddigital.com.au
www.oxforddigital.com.au
192.168.4.4
www.distance.vic.edu.au
lms.decvonline.vic.edu.au
connect.vic.edu.au
192.168.5.5
us.mineplex.com
stileapp.com
stileeducation.com
192.168.6.6

Actions #45

Updated by John K over 4 years ago

Jim Pingle wrote:

John K wrote:

What's the status here? Has Netgate been able to reproduce this issue?

Not that I have seen yet. We still need to find a combination of settings that reliably and repeatedly reproduces the issue.

Given Gavin Stewart's steps above, has Netgate finally been able to reproduce this issue now?

Actions #46

Updated by Gavin Stewart over 4 years ago

Gavin Stewart wrote:

I now have a minimal and repeatable set of steps to reproduce this.

Actually, I have revised the list of aliases to just one single host, still repeatable:

mail.bigpond.com

As per my previous instructions, with a clean install, and adding two host aliases for mail.bigpond.com (TEST1 and TEST2, applying changes between them), this is the entire clog output from /var/log/resolver.log (filterdns with -d 20):

Nov 27 06:18:13 pfSense filterdns:  Adding Action: pf table: TEST1 host: mail.bigpond.com
Nov 27 06:18:13 pfSense filterdns:      Adding host mail.bigpond.com
Nov 27 06:18:13 pfSense filterdns: Creating a new thread for action type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:18:13 pfSense filterdns: Creating a new thread for host mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns:      found address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns:          adding address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns: Change detected on host: mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns:  Awaking from the sleep for type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:18:20 pfSense filterdns:      Added pf address, table: TEST1 host: mail.bigpond.com address: 203.36.137.241
Nov 27 06:18:20 pfSense filterdns:  Updated pf table TEST1 host: mail.bigpond.com error: 0
Nov 27 06:18:24 pfSense filterdns: Received signal Hangup(1).
Nov 27 06:18:24 pfSense filterdns: merge_config: configuration reload
Nov 27 06:18:24 pfSense filterdns: Copied 1 actions to old
Nov 27 06:18:24 pfSense filterdns:  Adding Action: pf table: TEST1 host: mail.bigpond.com
Nov 27 06:18:24 pfSense filterdns:  Adding Action: pf table: TEST2 host: mail.bigpond.com
Nov 27 06:18:24 pfSense filterdns: Copied 2 actions to new
Nov 27 06:18:24 pfSense filterdns: Cleaning up action type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:18:24 pfSense filterdns: Loaded actions: 1 old and 1 new = 2 total
Nov 27 06:18:24 pfSense filterdns: Cleaning up previous actions
Nov 27 06:18:24 pfSense filterdns: Creating a new thread for action type: pf table: TEST2 hostname: mail.bigpond.com
Nov 27 06:18:24 pfSense filterdns:  Awaking from the sleep for hostname mail.bigpond.com (2)
Nov 27 06:18:24 pfSense filterdns:      found address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:23:23 pfSense filterdns:  Awaking from the sleep for hostname mail.bigpond.com (2)
Nov 27 06:23:24 pfSense filterdns:      found address 203.36.137.241 for host mail.bigpond.com

[2.4.4-RELEASE][root@pfSense.localdomain]/root: cat /var/etc/filterdns.conf
pf mail.bigpond.com TEST1
pf mail.bigpond.com TEST2
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -T show -t TEST1
   203.36.137.241
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -T show -t TEST2
[2.4.4-RELEASE][root@pfSense.localdomain]/root: 

As can be seen, filterdns doesn't ever add the resolved address to the second table, unless filterdns is killed and restarted, resulting in this addition to the log:

Nov 27 06:26:10 pfSense filterdns: Received signal Terminated(15).
Nov 27 06:26:10 pfSense filterdns: Cleaning up action type: pf table: TEST2 hostname: mail.bigpond.com
Nov 27 06:26:10 pfSense filterdns: Cleaning up action type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:26:10 pfSense filterdns: Waiting 2 seconds for threads to finish
Nov 27 06:26:10 pfSense filterdns:  Awaking from the sleep for hostname mail.bigpond.com (0)
Nov 27 06:26:10 pfSense filterdns: Cleaning up hostname mail.bigpond.com
Nov 27 06:26:10 pfSense filterdns:          removing address 203.36.137.241 from host mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns:  Adding Action: pf table: TEST1 host: mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns:      Adding host mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns:  Adding Action: pf table: TEST2 host: mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns: Creating a new thread for action type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns: Creating a new thread for action type: pf table: TEST2 hostname: mail.bigpond.com
Nov 27 06:26:35 pfSense filterdns: Creating a new thread for host mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:      found address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:          adding address 203.36.137.241 for host mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns: Change detected on host: mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:  Awaking from the sleep for type: pf table: TEST1 hostname: mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:      Added pf address, table: TEST1 host: mail.bigpond.com address: 203.36.137.241
Nov 27 06:26:36 pfSense filterdns:  Updated pf table TEST1 host: mail.bigpond.com error: 0
Nov 27 06:26:36 pfSense filterdns:  Awaking from the sleep for type: pf table: TEST2 hostname: mail.bigpond.com
Nov 27 06:26:36 pfSense filterdns:      Added pf address, table: TEST2 host: mail.bigpond.com address: 203.36.137.241
Nov 27 06:26:36 pfSense filterdns:  Updated pf table TEST2 host: mail.bigpond.com error: 0
[2.4.4-RELEASE][root@pfSense.localdomain]/root: cat /var/etc/filterdns.conf
pf mail.bigpond.com TEST1
pf mail.bigpond.com TEST2
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -T show -t TEST1
   203.36.137.241
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -T show -t TEST2
   203.36.137.241
[2.4.4-RELEASE][root@pfSense.localdomain]/root: 
Actions #47

Updated by Gavin Stewart over 4 years ago

I have a fix for this, and have created a pull request.

https://github.com/pfsense/FreeBSD-ports/pull/714

Actions #48

Updated by Jim Pingle over 4 years ago

  • Status changed from New to Pull Request Review
Actions #49

Updated by Luiz Souza over 4 years ago

  • Status changed from Pull Request Review to Feedback
  • % Done changed from 0 to 100

A fix based on Gavin's PR was committed, please let me know if the problem persists.

Thanks

Actions #50

Updated by Robert Gijsen over 4 years ago

Luiz Souza wrote:

A fix based on Gavin's PR was committed, please let me know if the problem persists.

Thanks

Maybe a stupic question, but as I don't have any git or build tools available within pfSense obviously, how can we test this? Does that mean we'd need to install the 2.5 nightly? Sidequestion, is there any ETA on 2.5 RTM?

Actions #51

Updated by Christian Ullrich over 4 years ago

  • Robert Gijsen wrote:

Maybe a stupic question, but as I don't have any git or build tools available within pfSense obviously, how can we test this? Does that mean we'd need to install the 2.5 nightly? Sidequestion, is there any ETA on 2.5 RTM?

In a (very large) nutshell:

$ git clone https://github.com/pfsense/freebsd-ports pfsense-ports
$ git clone -b RELENG_2_4_4 https://github.com/pfsense/freebsd-src pfsense-src
# poudriere ports -cp pfsense -m null -M $PWD/pfsense-ports
# poudriere jail -cj pfsense244 -m src=$PWD/pfsense-src -b
# echo ALLOW_UNSUPPORTED_SYSTEM=1 >> /usr/local/etc/poudriere.d/pfsense-make.conf
# poudriere bulk -j pfsense244 -p pfsense -z default net/filterdns
$ scp /usr/local/poudriere/data/packages/pfsense244-pfsense-default/All/filterdns-2.0_3.txz $WHEREVER
$ ssh $WHEREVER pkg install -f filterdns-2.0_3.txz

The ALLOW_UNSUPPORTED_SYSTEM line is necessary if the next line fails (on a FreeBSD 12 build system).

Actions #52

Updated by Christian Ullrich over 4 years ago

  • Luiz Souza wrote:

A fix based on Gavin's PR was committed, please let me know if the problem persists.

Confirmed. With rebuilt filterdns-2.0_3 on pfSense 2.4.4-p3, the tables are now populated correctly.

Actions #53

Updated by Luiz Souza over 4 years ago

  • Status changed from Feedback to Resolved
Actions #54

Updated by Jim Pingle over 4 years ago

  • Target version changed from 2.5.0 to 2.4.5
Actions #55

Updated by Robert Gijsen over 4 years ago

Luiz Souza wrote:

A fix based on Gavin's PR was committed, please let me know if the problem persists.

Thanks

I have compiled the package for our test-environment (huge thanks to Christian Ullrich for the info, I couldn't have done that without his help) and so far tables are populated as it should now.

Actions #56

Updated by Jim Pingle over 4 years ago

  • Status changed from Resolved to Feedback

Needs checked and/or tested again on 2.4.5 snapshots

Actions #57

Updated by Viktor Gurov about 4 years ago

Jim Pingle wrote:

Needs checked and/or tested again on 2.4.5 snapshots

tested on pfSense 2.4.5.a.20191220.1407

works as expected,
Resolved

Actions #58

Updated by Jim Pingle about 4 years ago

  • Status changed from Feedback to Resolved
Actions #59

Updated by Eduard Rozenberg about 4 years ago

Great to hear about the fix! Would have loved to see a 2.4.4 update with this fixed package, or even just a fixed filterdns package I could download and install on all my 2.4.4 setups, rather than waiting an unknown and growing number of months for 2.4.5 to be released.

As it is I'm spending a number of hours downloading and setting up a FreeBSD 11.2 virtual machine, installing git/poudriere, wastefully downloading multiple GB's of ports and source, and going through the filterdns package rebuild process. And hoping I'll have a fixed filterdns package at the end of this process.

This issue has been an ongoing pain for over a year and even trying to apply the fix is no carnival. But thanks again to the folks who actually dug into and resolved the issue.

Actions #60

Updated by Eduard Rozenberg about 4 years ago

Christian Ullrich wrote:

  • Robert Gijsen wrote:

Maybe a stupic question, but as I don't have any git or build tools available within pfSense obviously, how can we test this? Does that mean we'd need to install the 2.5 nightly? Sidequestion, is there any ETA on 2.5 RTM?

In a (very large) nutshell:
...

Thanks Christian for these instructions in the earlier post above. I'm not the only pfSense user who's not a FreeBSD and pfSense package build expert, so without his help I don't think any normal person would attempt this.

The following steps were required on a fresh FreeBSD 11.2 system/VM before following Christian's instructions.

$ pkg update -f
$ pkg install git
$ pkg install poudriere
$ mkdir /usr/local/poudriere
$ mkdir /usr/ports/distfiles

After doing these initial steps I ran through Christian's instructions and it worked fine to generate the package filterdns-2.0_3.txz which I installed on my various firewalls and things seem to be working fine. The poudriere jail command took a bunch of time (not sure how many hours on my underpowered VM), but it was done when I checked it the next day :).

Note that I did need to also run Christian's line containing ALLOW_UNSUPPORTED_SYSTEM even on my FreeBSD 11.2 VM.

I went to Status -> Filter Reload and clicked Reload Filter on each firewall after installling the new filterdns package.

I'm attaching the filterdns package I generated but of course you should never use someone else's binary package, and especially not on any security sensitive equipment :).

Actions #61

Updated by Eduard Rozenberg about 4 years ago

It appears a reboot was required on each firewall after updating the filterdns package to my custom built one (2.0_3). Without the reboot, the alias table was still not fully and correctly populated (i.e. there were still ip addresses missing from the list when running $ pfctl -T show -t my_alias_name). Hopefully the problem is well and truly solved now dammit :).

Actions #62

Updated by Eduard Rozenberg about 4 years ago

Still not working properly, at least a couple of IP's are still not populating in the table. Giving up for now, will wait for 2.4.5 and see if that somehow fixes things. Maybe my filterdns build didn't pick up the latest code fix or something, no idea.

Actions #63

Updated by Fabián Burbano about 4 years ago

Eduard Rozenberg wrote:

Still not working properly, at least a couple of IP's are still not populating in the table. Giving up for now, will wait for 2.4.5 and see if that somehow fixes things. Maybe my filterdns build didn't pick up the latest code fix or something, no idea.

Version 2.4.5 already has several RCs. I think it is safer to upgrade to the RC than to do such a process. At least in this case, due to the urgency. The final version should come out in a few days or maybe hours. I have three days using the RC and it works very well.

Actions #64

Updated by Eduard Rozenberg about 4 years ago

Fabián Burbano wrote:

Version 2.4.5 already has several RCs. I think it is safer to upgrade to the RC than to do such a process. At least in this case, due to the urgency. The final version should come out in a few days or maybe hours. I have three days using the RC and it works very well.

Thanks! My problem is definitely NOT resolved by my filterdns custom build. Having exactly the same issue. Will upgrade to 2.4.5 when it releases, hopefully soon. Got burned by at least 2-3 pfSense version upgrades in the past, so would like to minimize risk by waiting for 2.4.5 official release.

Actions #65

Updated by Robert Gijsen about 4 years ago

I also have to come back to my conclusion it was ok with the rebuild filterdns. While working better than before, tables are still not filled correctly when a FQDN is used multiple times in the aliasses.

Actions #66

Updated by Chris Poillion about 4 years ago

This bug still persists in Build 2.4.5.r.20200307.0900.
.

Actions #67

Updated by Gabriel Ribeiro almost 4 years ago

This bug still persists in Build 2.4.5 date:2020.04.09

Actions #68

Updated by Luiz Souza almost 4 years ago

How it was tested ? What was the result ? How it failed ?

Actions #69

Updated by Gabriel Ribeiro almost 4 years ago

This bug still persists in Build 2.4.5 date:2020.04.10

I can confirm my issue is the same as described by the other posters on this bug.

Logs show that filterdns claims to be doing the right thing - all expected alias entries (FQDN's, IP's, networks) show up in:
$ clog /var/log/resolver.log | grep "Adding Action"

But the alias table is incomplete, some (FQDN) addresses are missing:
$ pfctl -T show -t my_alias_name

There is no DNS resolution issue with any of the FQDN's - if I ping the FQDN's from the firewall their IP addresses are resolved.

Restarting the filter, re-saving the alias does not help

Actions #70

Updated by Eduard Rozenberg almost 4 years ago

Yep. Same issue. Today got locked out again out of all our sites. My workaround is to use a personal VPN to force my own IP to change, which then normally will allow me back in once the firewall re-evaluates the DNS name -> IP mapping in the hostnames alias. As others have speculated, issue may be linked to cases where there are multiple hostnames which evaluate to the same IP inside a pfSense alias. But I don't really have a firm idea or clue.

Actions #71

Updated by Donn Lasher almost 4 years ago

Same problem here - 2.4.5-RELEASE (amd64)

pfctl -T show -t my_remote
   71.237.XX.XX
   97.120.XX.XX
   184.100.XX.XX
cat /var/etc/filterdns.conf 
pf connection1.dyndns.org my_remote
pf connection2.dyndns.org my_remote
pf connection3.dyndns.org my_remote

My favorite part is the firewall log, showing the blocks.

    May 30 13:42:37    COMCAST      71.237.XX.XX:30614      96.65.XX.XX:21443    TCP:S

Wish I knew how to fix it.. the "Aliases Hostnames Resolve Interval" didn't help :(

Actions #72

Updated by Gavin Stewart almost 4 years ago

Donn Lasher wrote:

Same problem here - 2.4.5-RELEASE (amd64)

This is confirmed.

I am able to replicate the failure in a test VM, using my instructions in #9296#note-44

I will point out that I cannot replicate the failure with just one host alias as described in #9296#note-46

Actions #73

Updated by Gavin Stewart almost 4 years ago

Gavin Stewart wrote:

This is confirmed.

I am able to replicate the failure in a test VM, using my instructions in #9296#note-44

Just revising this statement, because the error isn't immediately reproducible with the instructions linked above. There is a distinct change in the symptoms, and it now appears to only relate to IP addresses in aliases, and not FQDNs.

I now have the following steps to reproducibly demonstrate failure:

  1. Firewall -> Aliases -> Import
  2. Alias Name "TEST1", import entire alias list below, Save, Apply Changes
  3. Diagnostics -> Tables -> "TEST1"
    + Note 2 entries, specifically the 192.168.1.1 address.
  4. Firewall -> Aliases -> Import
  5. Alias Name "TEST2", import entire alias list below, Save, Apply Changes
  6. Diagnostics -> Tables -> "TEST2"
    + Note table is the same as TEST1
  7. Alias Name "TEST3", import entire alias list below, Save, Apply Changes
  8. Diagnostics -> Tables -> "TEST3"
    + Note that 192.168.1.1 addresses does not appear.
  9. Kill the filterfdns process at the console, and restart manually:
    /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
    + Note that "TEST3" table is now properly populated.

List of aliases to import:

mail.bigpond.com
192.168.1.1

Notes
  • Test may be replicated over again by deleting all aliases and starting again.
  • Test appears to always require third alias to occur
  • This case also fails in the patched filterdns I have been using with 2.4.4-RELEASE-p3, so it would appear there is still an undiagnosed bug in filterdns, this ticket should probably be reopened.
Actions #74

Updated by Rob Shiras over 3 years ago

I just ran into this today. I was using IP addresses for the bookkeeper. She finally got a hostname with DynDNS.
So I went to Firewall>Aliases and changed her IP address into the hostname. I verified that the hostname resolves to her IP address. She connected once (with RDP) with no problems.
Subsequent tries fail. She gets the RDC dialog that says Remote Desktop can't connect to the remote computer for one of these reasons....
I add the IP back into the alias and she is again able to connect via RDP.
Interesting development here...My hostname works without issue. My hostname is a CNAME I added to my GoDaddy account, pointing to my public static IP.
Her hostname does not work. Her public IP is dynamic, and she uses DynDNS. What's the difference? They both resolve to a public IP correctly.

Actions #75

Updated by Luiz Souza over 3 years ago

Thanks for the detailed instructions Gavin.

I pushed a fix which should do the right thing in this case.

Please test the new version (filterdns-2.0_4) and let me know if the problem persists.

  • The new package is available with the 2.5 snapshots.
Actions #76

Updated by Gavin Stewart over 3 years ago

Luiz Souza wrote:

Please test the new version (filterdns-2.0_4) and let me know if the problem persists.

This problem persists unfortunately.

I have recompiled filterdns from a pull of the latest devel branch (55691996), and the problem remains while following the instructions in #9296#note-73

Actions #77

Updated by Brendon Baumgartner over 3 years ago

Should the status on this be changed? It says resolved.

Actions #78

Updated by Eduard Rozenberg over 3 years ago

Brendon Baumgartner wrote:

Should the status on this be changed? It says resolved.

Definitely not resolved. It's an unpredictably occuring issue that continues to terrorize admins.

There's related strange behavior - using an IP address that was part of an alias, still cannot access the network when I use that IP as part of an allow from source rule.

The only way I can deal with the issue is to use a VPN to change my own IP, and use the VPN IP in the dynamic FQDN specified in the alias.

Actions #79

Updated by Renato Botelho over 3 years ago

  • Status changed from Resolved to New
  • Target version changed from 2.4.5 to 2.5.0

Luiz, can you please take a look?

Actions #80

Updated by Anonymous over 3 years ago

  • Priority changed from High to Low
Actions #81

Updated by Max Knabe about 3 years ago

I just registered here to say that I believe I'm experiencing this exact bug (see https://forum.netgate.com/topic/159856/pass-rule-as-an-exception-to-a-reject-rule-doesn-t-match). If it's indeed the same bug, it should not be considered resolved, because I'm using the latest snapshot (2.5.0.a.20210113.2350) nor should it be considered low priority, because it essentially breaks the firewall functionality of a firewall OS. Both workarounds posted earlier (#9296#note-14 and #9296#note-16) did not work for me. Neither did reinstalling and importing the config.xml.

The only workaround that does actually work is running a script on a webserver that resolves lists of FQDNs and creating URL aliases in pfSense pointing to that script, which seems a bit ridiculous.

However, thanks to the people who are working on resolving this issue!

Actions #82

Updated by Viktor Gurov about 3 years ago

same issue on 2.5.0.a.20210120.1500

mixed alias entries:
- yandex.ru
- 1.2.3.4

# pfctl -t mix -Ts
   213.180.193.56

but if you create another one alias (any type) and click 'apply':

# pfctl -t mix -Ts
   1.2.3.4
   213.180.193.56

the issue seems to only occur when the alias is first created

Actions #83

Updated by Viktor Gurov about 3 years ago

  • Status changed from New to Confirmed
  • Affected Version changed from 2.4.4_2 to 2.5.0

see also #7209

Actions #84

Updated by Anonymous about 3 years ago

  • Target version changed from 2.5.0 to CE-Next
Actions #85

Updated by Robert Gijsen about 3 years ago

Wait wut? This got postponed AGAIN? This is a breaking issue for two years and a few days now, and still it's priority is 'low' and it gets postponed again? How much more info to reproduce do you actually need?

Sorry for my rant, but in all sincerity I, and I assume a lot of us, would like to know by now where this is going. I've been unable to upgrade pfSense for almost 2 years without breaking our setup completely because of this, which makes it an unsafe box however you want to explain it. What can we do more to help you troubleshoot or fix this?

Actions #86

Updated by Christian Ullrich about 3 years ago

Same sentiment here as Robert Gijsen's above.

Do we at least know whether the bug is in filterdns itself (generating wrong updates) or in pf (processing valid updates wrongly)? If it is filterdns's fault, I think the best way to proceed is to rewrite it. The way it works at the moment is to run one thread per host name and one thread per alias, so any reasonable setup quickly reaches into the hundreds of threads, all mostly waiting.

The last proposed fix (that I initially thought was working) was changing the way all these threads are communicating with each other, which is always the most difficult part to get right.

Actions #87

Updated by Gabriel Ribeiro over 2 years ago

same issue on 2.5.2-RELEASE - date 20211109

3 years...
Actions #88

Updated by Artur Mitrosz about 2 years ago

I use 2.5.2-RELEASE (amd64) - Jul, 02 15:33:00 EDT 2021 - with exactly the same problem.
After killing filterdns (pkill filterdns) and reloading filter (Status -> Filter Reload) tables are updated correctly, but only for some minutes (probably until automatic refresh occurs), when the same alias table drop number of addresses from eg. 135 to 82. Even after that count of addresses in this example alias table changes between from about 76 up to 90 (when it should has above 130).
I post sample resolver logs here - it seems to have some errors, but my knowledge is too low to resolve this.


Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: watson.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: watson.microsoft.com address: 20.42.73.29
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: watson.microsoft.com address: 13.89.179.12 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: watson.microsoft.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: secure.meetup.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: secure.meetup.com address: 151.101.114.217
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: secure.meetup.com address: 151.101.14.217 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: secure.meetup.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.55 for host shredder.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.21.226 for host ocsp2.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.20.226 for host ocsp2.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:15e2 for host ocsp2.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:14e2 for host ocsp2.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.176 for host officepreviewredir.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.20.226 for host secure.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.21.226 for host secure.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:14e2 for host secure.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:15e2 for host secure.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.42.12 for host d.docs.live.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.121 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.109.88.121 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.83 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.109.88.83 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.46 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.109.88.46 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.120 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.109.88.120 for host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.109.76.98 from host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.109.76.122 from host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.109.76.123 from host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.109.76.99 from host nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: nleditor.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.88.120
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.88.46
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.88.83
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.88.121
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.76.98 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.76.122 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.76.123 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: nleditor.osi.office.net address: 52.109.76.99 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: nleditor.osi.office.net error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 51.116.48.199 for host informationprotection.hosting.portal.azure.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 20.82.250.189 for host smartscreen.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 23.197.104.63 for host officecdn.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a02:26f0:7100:1a1::6bb for host officecdn.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a02:26f0:7100:1b4::6bb for host officecdn.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 204.79.197.200 for host c.bing.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.42.12 for host docs.live.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.20.226 for host crl.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.18.21.226 for host crl.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:15e2 for host crl.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2606:4700::6812:14e2 for host crl.globalsign.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 162.125.248.18 for host dropbox.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:100:6040:18::a27d:f812 for host dropbox.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.76.0 for host wikipedia.firstpartyapps.oaspapps.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2.16.172.58 for host isrg.trustid.ocsp.identrust.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 92.123.189.128 for host isrg.trustid.ocsp.identrust.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a02:26f0:64::215:f2c5 for host isrg.trustid.ocsp.identrust.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a02:26f0:64::215:f2bb for host isrg.trustid.ocsp.identrust.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.88.132 for host protection.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 184.31.81.167 for host videocontent.osi.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.81.113.223 for host apps.skypeassets.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.76.0 for host excelbingmap.firstpartyapps.oaspapps.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 34.252.74.1 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.31.48.193 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 46.137.171.215 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a05:d018:76c:b685::de70 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a05:d018:76c:b680::de70 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2a05:d018:76c:b684::de70 for host sourcedeliveries.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.43.88.59 for host meechum.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 50.112.200.172 for host meechum.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 34.209.107.23 for host meechum.netflix.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.127.185.23 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 40.127.185.23 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 65.52.144.94 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 65.52.144.94 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.113.92.244 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 40.113.92.244 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.113.83.221 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 40.113.83.221 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.46.33.89 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 104.46.33.89 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 65.52.151.72 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 65.52.151.72 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 137.116.199.193 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 137.116.199.193 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 104.40.156.160 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 104.40.156.160 for host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.115.58.225 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 23.97.209.231 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 23.97.169.223 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.113.92.113 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.127.133.56 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 23.102.24.87 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.127.179.193 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.115.10.186 from host dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: SKYPE_HOSTS hostname: dr.skype.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 104.40.156.160
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 137.116.199.193
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 65.52.151.72
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 104.46.33.89
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.113.83.221
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.113.92.244
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 65.52.144.94
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.127.185.23
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.115.58.225 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 23.97.209.231 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 23.97.169.223 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.113.92.113 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.127.133.56 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 23.102.24.87 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.127.179.193 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: dr.skype.net address: 40.115.10.186 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table SKYPE_HOSTS host: dr.skype.net error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]: failed to resolve host directory.services.live.com will retry later again.
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.76.0 for host peoplegraph.firstpartyapps.oaspapps.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 35.214.201.112 for host www.eztitles.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.213.44 for host signup.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.246.44 for host signup.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:1ec:bdf::44 for host signup.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:1ec:46::44 for host signup.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.69.106.216 for host dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 13.69.106.216 for host dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.236.186.217 from host dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: dc.services.visualstudio.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: dc.services.visualstudio.com address: 13.69.106.216
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: dc.services.visualstudio.com address: 52.236.186.217 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: dc.services.visualstudio.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.246.44 for host mem.gfx.ms
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.213.44 for host mem.gfx.ms
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:1ec:bdf::44 for host mem.gfx.ms
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2620:1ec:46::44 for host mem.gfx.ms
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.79.197.34 for host pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 40.79.197.34 for host pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.168.117.169 from host pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.113.130.243 for host myanalytics.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: SKYPE_HOSTS hostname: pipe.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: pipe.skype.com address: 40.79.197.34
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: pipe.skype.com address: 52.168.117.169 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table SKYPE_HOSTS host: pipe.skype.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.113.199.175 for host api.asm.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 23.197.104.24 for host omextemplates.content.office.net
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 40.76.42.76 for host myanalytics-gcc.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.174.193.75 for host get.skype.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 20.44.10.123 for host mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 20.44.10.123 for host mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 40.79.197.35 from host mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: SKYPE_HOSTS hostname: mobile.pipe.aria.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: SKYPE_HOSTS host: mobile.pipe.aria.microsoft.com address: 20.44.10.123
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: SKYPE_HOSTS host: mobile.pipe.aria.microsoft.com address: 40.79.197.35 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table SKYPE_HOSTS host: mobile.pipe.aria.microsoft.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.107.6.162 for host officespeech.platform.bing.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.76.22 for host testconnectivity.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.111.243.9 for host augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 52.111.243.9 for host augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.111.243.8 from host augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: augloop.office.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: augloop.office.com address: 52.111.243.9
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: augloop.office.com address: 52.111.243.8 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: augloop.office.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 13.69.106.89 for host dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  adding address 13.69.106.89 for host dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:                  removing address 52.236.186.217 from host dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]: Change detected on host: dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:  Awaking from the sleep for type: pf table: OFFICE365_REQUIRED hostname: dc.applicationinsights.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          Added pf address, table: OFFICE365_REQUIRED host: dc.applicationinsights.microsoft.com address: 13.69.106.89
Jan 17 00:13:23 rtr1 filterdns[62542]:          FAILED to remove pf address, table: OFFICE365_REQUIRED host: dc.applicationinsights.microsoft.com address: 52.236.186.217 error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:  Updated pf table OFFICE365_REQUIRED host: dc.applicationinsights.microsoft.com error: -1
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 52.109.20.9 for host office.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 2603:1030:800:5::bfee:ac7d for host office.microsoft.com
Jan 17 00:13:23 rtr1 filterdns[62542]:          found address 51.116.60.32 for host management.azure.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www5.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www5.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www3.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www3.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www7.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www7.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www4.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www4.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.79.80.20 for host www6.atomicorp.com
Jan 17 00:13:24 rtr1 filterdns[62542]:          found address 51.161.13.101 for host www6.atomicorp.com

Actions #89

Updated by Viktor Gurov about 2 years ago

  • Related to Bug #12708: Alias with non-resolving FQDN entry breaks underlying PF table added
Actions #90

Updated by D D about 2 years ago

I thought this would have been fixed with 2.6. I had to reenable the cron workaround. Oh well.

Actions #91

Updated by → luckman212 about 2 years ago

Should this really be a low priority?
Seems like improper alias tables could pose a security risk.

Also, progress for this bug still says 100%
And there's no pfSense+ target version set.

Can someone please dust this off? it's affected me too, hope it can get some attention, seems there are reliable repro steps here.

Actions #92

Updated by Chris K about 2 years ago

Can you guys try out below workaround for max threads per process? I have been suffering now for weeks with this issue, rendering my firewall unreliable and useless, issues with VoIP etc. I also reverted back from 2.6 to 2.5.2, no change. But it seems, that for me this workaround solved the issue- it remains open if that is just temporary... I will now try upgrading back to 2.6.0 and retest.

[[https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html]]
Let me know your results. This issue is very annoying.

Actions #93

Updated by Chris K about 2 years ago

As a Sidenote: after updating to 2.6.0 a once working ruleset completely broke. I have now restored the backup and again works fine on 2.5.2. What a mess.

Actions #94

Updated by Charlie Blalock almost 2 years ago

Been fighting this issue on 2.5 and 2.4.5 and I am talking about using only 1 DNS entry in the Alias to a Dynamic DNS on several firewalls - intermittent with various ones and difficult to pin down. Will try a different firewall at one of the site and see if that occurs; another 'sense.

Actions #95

Updated by Tom Huerlimann almost 2 years ago

Same here, plase fix it.

Actions #96

Updated by Marcos M almost 2 years ago

I was able to (unreliably) reproduce this on latest 22.05 snapshot. I then exited filterdns and started it with verbosity 99; I was not able to reproduce the issue afterwards. I tried to reproduce the issue following the different steps in the comments. If someone can reliably reproduce the issue, feel free to apply the attached patch to increase the filterdns verbosity and provide the logs.

Actions #97

Updated by Eduard Rozenberg almost 2 years ago

I added the setting from https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html and will give it some time to see if this fixes the issue.

Actions #98

Updated by Marcos M almost 2 years ago

  • Subject changed from Rule / Alias FQDN-Resolution broken to Alias content is sometimes incomplete when mixing FQDN and IP address
  • Assignee deleted (Luiz Souza)
  • % Done changed from 100 to 0
  • Plus Target Version set to Plus-Next
Actions #99

Updated by Kris Phillips over 1 year ago

It seems this issue has gotten worse somewhere along the line similar to how others are describing it. Tables now load empty rather than only partially loading.

Also there is a related bug likely tied to the same sanity checks (or lack of):
https://redmine.pfsense.org/issues/13282

Actions #100

Updated by Steve Wheeler over 1 year ago

  • Assignee set to Reid Linnemann
Actions #101

Updated by Reid Linnemann over 1 year ago

I'm having a crack at this issue now. Is everyone experiencing this issue using unbound as a resolver by chance?

Actions #102

Updated by Kris Phillips over 1 year ago

Reid Linnemann wrote in #note-101:

I'm having a crack at this issue now. Is everyone experiencing this issue using unbound as a resolver by chance?

Since unbound is the default, I'd imagine it's related. I have only seen this crop up with unbound being used as DNS for the firewall first, with remote second. Will need to confirm next time I can reproduce it.

Actions #103

Updated by Reid Linnemann over 1 year ago

There are several things I've noted about how aliases and filterdns work that - if they aren't directly related to this problem - certainly don't help. I'm going to start by going down those rabbit holes and see where they take me.

I was suspecting unbound caching behavior could get in the way if filterdns was issuing ANY queries, but I have since ruled that out - filterdns uses getaddrinfo, which makes A and AAAA queries.

Actions #104

Updated by Bob Smith over 1 year ago

Had to create an account just to leave a note regarding this issue

We host a text file at https://www.mydomainurl.com/list/ip.txt (not the real domain)
This is our "traveling webgui access list" - its literally a flat txt file with 10-15 ip addresses listed one per line
We have the URL setup in pfsense as an Alias URL Table
We have cron set to update urltables every 2mins with "now forceupdate"
When traveling or at a remote site and needing temp access to a router, we will update this txt list via ftp and cron will pull a new copy
When the new copy is pulled, the updated IP we added to the list is now part of our "webgui access rule"
Now we can login to the router from that new IP we added, and when done we'll remove the IP from the list and upload a new copy via ftp
Log file shows every 2mins URL table updates with either "no changes" or "1 address added/removed"

Earlier today, for some reason the firewall stopped updating this alias URL table???
The list that shows is "stuck" and has our 9 IP addresses listed
Attempts to delete and recreate it yield the EXACT SAME 9 IP addresses again with NO changes or updates.
Attempts to add a new list, named differently, which is added as a new alias and new rule, which is even hosted at a different URL (like github) yield the EXACT SAME ORIGINAL 9 IP addresses!!?!
Mousing over the existing alias rule and the new alias rule - the same 9 IP addresses are listed, but the URLs shown at the top of each list are different. Very weird.
If you browse up each URL in the web browser, they show different correct new lists (more than 9 IPS listed)

  • Rebooting the router does not help
  • Updating the tunables with the kern.threads.max_threads_per_proc = 4096 does not help (even post reboot)
  • pkill filterdns and reloading it does not help

And here's the kicker: we have approx 70 pfsense boxes we run, all on 2.6 or 22.01 (either virtual or purchased from Netgate if a physical box), and all 70 have the original list stuck at the same point with the same 9 IP addresses from the last time they all updated. I've logged into most all of them the past 4hrs and can confirm they all show the list stuck at the same point despite the logs showing cron running, confirming it checks the URL and confirms there are "no changes" despite me updating the URL list via ftp to 10+ IP addresses.

I have tested updating the aliases, URLs, creating new ones, new rules etc as described above on at least 4 of them so far and they all exhibit the same behavior.

Creating standard aliases with individual host IP entries works fine, which is how I've had to set all of them currently.

We have had this setup for the past 3+ yrs with ZERO issues and pfsense for us since 2012 has been extremely dependable and stable, but today, not sure what the heck happened? Seems like this specific issue is ongoing and now bleeding into 2.6 and 22.01... Also wanted to make it known this has a very real impact regarding sense of stability for us.

Open to suggestions on how to fix.

Actions #105

Updated by Bob Smith over 1 year ago

UPDATE: Tinkering some more this morning. Found out that if I make a new alias URL table, point it to a new URL list that starts with an invalid IP like 10.2.3.4, let it update that IP once, then I can add subsequent IPs to the list and the list now updates normally... now in the process of updating each firewall with the new alias/list and removing the old alias/list.

Actions #106

Updated by Reid Linnemann over 1 year ago

Bob, thank you for your detailed report. Can you confirm for me that all of the entries in the hosted list are IPs, and no FQDNs are used?

Actions #107

Updated by Marco Jäger over 1 year ago

Hi all,
i think this issue is solved in the version 2.6.0. I have 2 diffrent pfsense. One is on the verison 2.4.4-P1 (FreeBSD 11.2) and the other is on 2.6.0 (FreeBSD 12.3). If i compare the file /usr/local/sbin/filterdns from version 2.4.4-P1 with the version 2.6.0 i can see a file size diffrent.

The size of the file on 2.4.4-P1 is 36.120 B
The size of the file on 2.6.0 is 36.728 B

File Compare

The new file is bigger than the older one.

My test:

1) Connect with ssh to the pfsense with the version 2.6.0 and create a backup from the file /usr/local/sbin/filterdns
2) Get the PID (e.g. ps aux | grep filterdns) from filterdns and kill it with kill [PID]
3) Load the filterdns file from the pfsense with the version 2.4.4-P1 to the pfsense with the version 2.6.0
4) Making sure that the file has correct permissions/owner/group
5) Start the filterdns again with the webpage (System > Filter Reload)
6) Making sure that the process filterdns runs again (again ps aux | grep filterdns)
7) Now create a new alias and enter a FQDN
8) Check if the new created alias can be found at Diagnostics > Tables
9) Select it and check if it is filled with the ip address of the FQDN.

After this test i saw that when i use the filterdns from version 2.4.4-P1 on a pfsense with the version 2.6.0 the table of the new alias is not filled with the ip address. To fix that you need to kill the filterdns again (e.g. step 2) and step 5)). Than it will show the ip address in the table.
Broken Alias

In the second test i used the filterdns from the version 2.6.0 on the pfsense with the version 2.4.4-P1.
The steps:

1) Connect with ssh to the pfsense with the version 2.4.4-P1 and create a backup from the file /usr/local/sbin/filterdns
2) Get the PID (e.g. ps aux | grep filterdns) from filterdns and kill it with kill [PID]
3) Load the filterdns file from the pfsense with the version 2.6.0 to the pfsense with the version 2.4.4-P1
4) Making sure that the file has correct permissions/owner/group
5) Start the filterdns again with the webpage (System > Filter Reload)
6) Making sure that the process filterdns runs again (again ps aux | grep filterdns)
7) Now create a new alias and enter a FQDN
8) Check if the new created alias can be found at Diagnostics > Tables
9) Select it and check if it is filled with the ip address of the FQDN.

After creating a new alias with a FQDN it will apprear immediately in the Tables list.

Working Alias

So my suggestion is, that someone fixed this bug in the new version of the filterdns. I will upload the new filterdns here to this ticket so you can try it by yourself. I changed it now on our productive pfsense with the version 2.4.4-P1 and no issues so far.

Actions #108

Updated by Reid Linnemann over 1 year ago

Some of the issues with FQDNs are better with 2.6/2.7.0-development and 22.05, but there are still very real problems that seem to stem from filterdns. I have a reliable way to generate incomplete tables at this point that may shed some light on what's going on.

Actions #109

Updated by Bob Smith over 1 year ago

@Reid - per your previous question - yes our entire list is only IP addresses with a #comment after each address. No FQDNs in the list at all. The alias itself is a URL table with our FQDN: https://www.mydomainurl.com/list/ip.txt (not the real domain)

example of what the list is
0.0.0.0 #comment
1.2.3.4 #comment
etc.
etc.

Actions #110

Updated by Jim Pingle over 1 year ago

  • Priority changed from Low to Normal
  • Target version changed from CE-Next to 2.7.0
  • Plus Target Version changed from Plus-Next to 22.11
Actions #111

Updated by Reid Linnemann over 1 year ago

I think I have this pretty well nailed down to a race on initial thread creation for added aliases where the created thread is not waiting on a condvar before the routine that spawned threads cycles through them and signals them, which leads to the initial run through the action routine that adds the address to the alias table being delayed by the filterdns interval (5 minutes). I believe I will have a fix for snaps soon.

Actions #112

Updated by Reid Linnemann over 1 year ago

  • Status changed from Confirmed to Feedback
  • % Done changed from 0 to 100
Actions #113

Updated by Reid Linnemann over 1 year ago

  • Related to Bug #13067: Resolve interval for ``filterdns`` may not match the configured value added
Actions #114

Updated by Azamat Khakimyanov over 1 year ago

Tested on 22.05, 2.7-CE-DEV and 22.11-DEV (built on Fri Oct 07 15:14:37 UTC 2022)

I created different aliases:
- nested aliases
- big aliases with FQNDs and IPs and another aliases inside
and did test described in https://redmine.pfsense.org/issues/9296#note-44

There were no problem at all on 22.05 BUT on 2.7 and 22.11 when I did test https://redmine.pfsense.org/issues/9296#note-44 TEST2 alias wasn't complete immediately after I created it ('test2_alias.png'). After 1-2 minutes or pressing Filter Reload 2-3 times, TEST2 alias became complete.

I would retest it again on 2.7 and 22.11 when these versions become stable releases.

Actions #115

Updated by Jim Pingle over 1 year ago

  • Plus Target Version changed from 22.11 to 23.01
Actions #116

Updated by Brian M over 1 year ago

I have the same issue. Mixing FQDN and IP addresses caused me hours of frustration why various rules were not working. Finally seeing this bug report and the reported work-around of changing the timeout for Alias resolution worked.

I am using version:
2.6.0-RELEASE (amd64)
built on Mon Jan 31 19:57:53 UTC 2022
FreeBSD 12.3-STABLE

Actions #117

Updated by Reid Linnemann over 1 year ago

Brian M wrote in #note-116:

I have the same issue. Mixing FQDN and IP addresses caused me hours of frustration why various rules were not working. Finally seeing this bug report and the reported work-around of changing the timeout for Alias resolution worked.

That's a really hit-and-miss workaround, I fixed some very troublesome thread synchronization issues in filterdns that were the root cause. If you can update to 2.7.0-devel I'd really recommend it, especially if you're making extensive use of FQDN aliases.

Actions #118

Updated by Jim Pingle over 1 year ago

  • Subject changed from Alias content is sometimes incomplete when mixing FQDN and IP address to Alias content is sometimes incomplete when an alias contains both FQDN and IP address entries

Updating subject for release notes.

Actions #119

Updated by Alexey Ab about 1 year ago

I am testing https://snapshots.netgate.com/amd64/pfSense_master/installer/pfSense-CE-2.7.0-DEVELOPMENT-amd64-20221216-0600.iso.gz

There are same problems as in v2.4.5p1.

1) Add following domains to alias (conventional names and ip's):
aa.dyndns.org (2.2.2.2)
bb.dyndns.org (3.3.3.3)
cc.dyndns.org (4.4.4.4)
dd.dyndns.org (5.5.5.5)

2) Table will be correctly populated.

3) Change aa.dyndns.org to 3.3.3.3 in dyndns (duplicate of bb.dyndns.org), wait.

4) Table will become correct (3 entries, duplicates removed).

5) Change aa.dyndns.org back to 2.2.2.2 in dyndns, wait.

6) Table will not be correct anymore. It will miss 3.3.3.3 entry no matter how long you wait.

7) Clear table using red "Empty table" button in Diagnostic->Tables

8) Table will remain clear and won't be filled anymore, until restart.

It looks like incorrect caching of table entries.

Adding new (ADDR_NEW) and removing old (ADDR_OLD) address from pf table on dns update would work if duplicates were not removed from table, and won't work if duplicated addresses exist in different dns records, and then removed in pf table.

Allowing user to perform "Empty table" which brings tables to inconsistent state until firewall restart is also a strange solution.

And I have to agree with others - the lack of proper testing of such basic functionality, and not fixing it for years is not about "world's most trusted open source network security solution"...

Actions #120

Updated by Marcos M about 1 year ago

  • Status changed from Feedback to Confirmed

Another potentially related issue:
Editing an entry within an alias when that alias has been included within another alias (i.e. editing a nested alias) will result in the alias existing in rules.debug but not actually loaded in pf:

[23.01-BETA][root@gw]/root: grep '_alias' /tmp/rules.debug
table <_alias1_> persist
_alias1_ = "<_alias1_>" 
table <_alias2_> {   5.5.5.5 }
_alias2_ = "<_alias2_>" 
[23.01-BETA][root@gw]/root: pfctl -t _alias1_ -Ts
   5.5.5.5
[23.01-BETA][root@gw]/root: pfctl -t _alias2_ -Ts
pfctl: Unknown error: -1.

Regarding the comment #note-119, I can confirm the same behavior. Here's the filterdns log (reversed):

Dec 18 11:39:08     filterdns     35638     [100800] ("filterdns.c":507 host_dns()): found address 2.2.2.2 for host atest.domain.net
Dec 18 11:39:08     filterdns     35638     [100801] ("filterdns.c":507 host_dns()): found address 3.3.3.3 for host btest.domain.net
Dec 18 11:39:08     filterdns     35638     [100800] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname atest.domain.net (1)
Dec 18 11:39:08     filterdns     35638     [100801] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname btest.domain.net (1)
Dec 18 11:36:08     filterdns     35638     [100801] ("filterdns.c":507 host_dns()): found address 3.3.3.3 for host btest.domain.net
Dec 18 11:36:08     filterdns     35638     [100800] ("filterdns.c":507 host_dns()): found address 2.2.2.2 for host atest.domain.net
Dec 18 11:36:08     filterdns     35638     [100801] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname btest.domain.net (1)
Dec 18 11:36:08     filterdns     35638     [100800] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname atest.domain.net (1)
Dec 18 11:33:08     filterdns     35638     [100800] ("filterdns.c":507 host_dns()): found address 2.2.2.2 for host atest.domain.net
Dec 18 11:33:08     filterdns     35638     [100801] ("filterdns.c":507 host_dns()): found address 3.3.3.3 for host btest.domain.net
Dec 18 11:33:08     filterdns     35638     [100800] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname atest.domain.net (1)
Dec 18 11:33:08     filterdns     35638     [100801] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname btest.domain.net (1)
Dec 18 11:30:09     filterdns     35638     [100801] ("filterdns.c":507 host_dns()): found address 3.3.3.3 for host btest.domain.net
Dec 18 11:30:09     filterdns     35638     [100799] ("filterdns.c":141 check_action()): Updated pf table test anchor (null) host: atest.domain.net error: 0
Dec 18 11:30:09     filterdns     35638     [100799] ("tables.c":136 table_update()): Removed pf address, table: test anchor: (null) host: atest.domain.net address: 3.3.3.3
Dec 18 11:30:09     filterdns     35638     [100799] ("tables.c":136 table_update()): Added pf address, table: test anchor: (null) host: atest.domain.net address: 2.2.2.2
Dec 18 11:30:09     filterdns     35638     [100799] ("filterdns.c":107 check_action()): Awaking from the sleep for type: pf table: test hostname: atest.domain.net
Dec 18 11:30:09     filterdns     35638     [100800] ("filterdns.c":620 check_hostname()): Change detected on host: atest.domain.net
Dec 18 11:30:09     filterdns     35638     [100800] ("filterdns.c":456 addr_del()): removing address 3.3.3.3 from host atest.domain.net
Dec 18 11:30:09     filterdns     35638     [100800] ("filterdns.c":434 addr_add()): adding address 2.2.2.2 for host atest.domain.net
Dec 18 11:30:09     filterdns     35638     [100800] ("filterdns.c":507 host_dns()): found address 2.2.2.2 for host atest.domain.net
Dec 18 11:30:09     filterdns     35638     [100801] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname btest.domain.net (1)
Dec 18 11:30:09     filterdns     35638     [100801] ("filterdns.c":507 host_dns()): found address 3.3.3.3 for host btest.domain.net
Dec 18 11:30:09     filterdns     35638     [100800] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname atest.domain.net (1)
Dec 18 11:30:09     filterdns     35638     [100801] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname btest.domain.net (1)
Dec 18 11:30:09     filterdns     35638     [100800] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname atest.domain.net (1)
Dec 18 11:30:09     filterdns     35638     [100795] ("filterdns.c":300 action_del()): Cleaning up action type: pf table: test hostname: btest.domain.net
Dec 18 11:30:09     filterdns     35638     [100795] ("filterdns.c":300 action_del()): Cleaning up action type: pf table: test hostname: atest.domain.net
Dec 18 11:30:09     filterdns     35638     [100795] ("filterdns.c":281 action_add()): Adding Action: pf table: test host: btest.domain.net
Dec 18 11:30:09     filterdns     35638     [100795] ("filterdns.c":281 action_add()): Adding Action: pf table: test host: atest.domain.net
Dec 18 11:29:11     filterdns     35638     [100798] ("filterdns.c":141 check_action()): Updated pf table test anchor (null) host: btest.domain.net error: 0
Dec 18 11:29:11     filterdns     35638     [100798] ("tables.c":136 table_update()): Added pf address, table: test anchor: (null) host: btest.domain.net address: 3.3.3.3
Dec 18 11:29:11     filterdns     35638     [100798] ("filterdns.c":107 check_action()): Awaking from the sleep for type: pf table: test hostname: btest.domain.net
Dec 18 11:29:11     filterdns     35638     [100801] ("filterdns.c":620 check_hostname()): Change detected on host: btest.domain.net
Dec 18 11:29:11     filterdns     35638     [100801] ("filterdns.c":434 addr_add()): adding address 3.3.3.3 for host btest.domain.net
Dec 18 11:29:11     filterdns     35638     [100801] ("filterdns.c":507 host_dns()): found address 3.3.3.3 for host btest.domain.net
Dec 18 11:29:10     filterdns     35638     [100799] ("filterdns.c":141 check_action()): Updated pf table test anchor (null) host: atest.domain.net error: 0
Dec 18 11:29:10     filterdns     35638     [100799] ("tables.c":136 table_update()): Added pf address, table: test anchor: (null) host: atest.domain.net address: 3.3.3.3
Dec 18 11:29:10     filterdns     35638     [100799] ("filterdns.c":107 check_action()): Awaking from the sleep for type: pf table: test hostname: atest.domain.net
Dec 18 11:29:10     filterdns     35638     [100800] ("filterdns.c":620 check_hostname()): Change detected on host: atest.domain.net
Dec 18 11:29:10     filterdns     35638     [100800] ("filterdns.c":434 addr_add()): adding address 3.3.3.3 for host atest.domain.net
Dec 18 11:29:10     filterdns     35638     [100800] ("filterdns.c":507 host_dns()): found address 3.3.3.3 for host atest.domain.net
Dec 18 11:29:10     filterdns     35638     [100801] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname btest.domain.net (1)
Dec 18 11:29:10     filterdns     35638     [100800] ("filterdns.c":649 check_hostname()): Awaking from the sleep for hostname atest.domain.net (1)
Dec 18 11:29:10     filterdns     35638     [100795] ("filterdns.c":675 check_hostname_create()): Creating a new thread for host btest.domain.net
Dec 18 11:29:10     filterdns     35638     [100795] ("filterdns.c":675 check_hostname_create()): Creating a new thread for host atest.domain.net
Dec 18 11:29:10     filterdns     35638     [100795] ("filterdns.c":177 action_create()): Creating a new thread for action type: pf table: test hostname: atest.domain.net
Dec 18 11:29:10     filterdns     35638     [100795] ("filterdns.c":177 action_create()): Creating a new thread for action type: pf table: test hostname: btest.domain.net
Dec 18 11:29:10     filterdns     35638     [100795] ("filterdns.c":376 host_add()): Adding host btest.domain.net
Dec 18 11:29:10     filterdns     35638     [100795] ("filterdns.c":281 action_add()): Adding Action: pf table: test host: btest.domain.net
Dec 18 11:29:10     filterdns     35638     [100795] ("filterdns.c":376 host_add()): Adding host atest.domain.net
Dec 18 11:29:10     filterdns     35638     [100795] ("filterdns.c":281 action_add()): Adding Action: pf table: test host: atest.domain.net

This results in the alias missing 3.3.3.3 since it was correctly removed:

("filterdns.c":456 addr_del()): removing address 3.3.3.3 from host atest.domain.net

but not re-added afterwards:

("filterdns.c":507 host_dns()): found address 3.3.3.3 for host btest.domain.net

Actions #121

Updated by Marcos M about 1 year ago

  • Subject changed from Alias content is sometimes incomplete when an alias contains both FQDN and IP address entries to Using FQDN or Alias entries within an alias can result in incomplete table contents for that alias.
Actions #122

Updated by Jim Pingle about 1 year ago

  • Subject changed from Using FQDN or Alias entries within an alias can result in incomplete table contents for that alias. to Using FQDN or Alias entries within an alias can result in incomplete table contents for that alias
Actions #123

Updated by Reid Linnemann about 1 year ago

  • Subject changed from Using FQDN or Alias entries within an alias can result in incomplete table contents for that alias to Alias content is sometimes incomplete when an alias contains both FQDN and IP address entries
Actions #124

Updated by Jim Pingle about 1 year ago

  • Status changed from Confirmed to Feedback
  • Start date deleted (01/30/2019)

Lets keep this issue for just the stated problem here and ensure that any potentially related problems have their own distinct Redmine issue(s) if they don't already. There isn't going to just be one "fix filterdns" Redmine issue, we need each potential problem scenario and test case stated separately.

Actions #125

Updated by Reid Linnemann about 1 year ago

  • Start date set to 01/30/2019

The most recent comments above identify problems in filterdns that are fundamentally different in nature. I am opening two new issues to cover filterdns bugs involving multiple references to addresses in a table and failure to rectify the filter tables with modelled tables when modified via a sideband channel (e.g. clearing the table via pfctl), and I'll continue work on these. I will also link them to this issue.

Actions #126

Updated by Reid Linnemann about 1 year ago

  • Related to Bug #13792: Filterdns assumes sets of resolved addresses for each hostname are nonintersecting added
Actions #127

Updated by Reid Linnemann about 1 year ago

  • Related to Bug #13793: filterdns does not reconcile modelled tables with the current state of filter tables added
Actions #128

Updated by Reid Linnemann about 1 year ago

  • Start date deleted (01/30/2019)
Actions #129

Updated by Jim Pingle about 1 year ago

  • Status changed from Feedback to Resolved

The issue here related to the subject appears to be OK, and the other related issues have been spun off into their own Redmine issues.

We can reopen/revisit if needed, but make sure it's related to the subject and not one of the other similar Redmine issues for aliases/filterdns.

Actions #130

Updated by Robert Gijsen 7 months ago

Unfortunately, the exact thing happened again in 2.7.0 for us over the weekend. We use an external spamfilter where mail from our customers flows through. We have an alias for their DNS records, and some of them are in by IP as well (as they might change their DNS records over time). Last saturday mail stopped coming in. And looking at the tables, we had the same issue again; one IP was in as IP directly, and as FQDN. And the resulting IP was just not in the table for the alias.

Now this is only the first time we've had this happen in 2.7.0, but so far I consider this issue still not fixed. Are there any logs I can / should paste?

[edit] In the meanwhile I got closer to the issue; in the weekend our spamfilter changed one DNS record to an IP we had in that same alias statically. So, we had x.x.x.x statically, and some.dns.entry which now resolves to x.x.x.x as well. Result: duplicate entry, which was removed in the table.

This is an issue that's described extensively in this bugtracker thread, and as per 2.7.0 I still consider it not solved.

[edit2]
it's reproducible as well:

- Create an alias
- add 1.1.1.1
- add 8.8.8.8
- add a dns entry you created, pointing to 1.1.1.1, ie pfsensetest.domain.com
- monitor the table-entry for the alias, all will be ok
- now change the DNS entry from 1.1.1.1 to 8.8.8.8 and wait for it to be replicated
- in my setup, 1.1.1.1 got deleted from the table. So while 8.8.8.8 is in there 'twice' now, and 1.1.1.1 only once, it's not there anymore.
- killing filterdns and reloading filters repopulates the tables correctly it seems.

Tried 3 times, all with the same results.

Actions

Also available in: Atom PDF