https://redmine.pfsense.org/https://redmine.pfsense.org/favicon.ico?16780521162018-08-09T09:12:36ZpfSense bugtrackerpfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=376102018-08-09T09:12:36ZJim Pingle
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul><p>I had this happen on my edge firewall which is running a snapshot from earlier this week. The pf tables with entries managed by <code>filterdns</code> were empty when it failed, even entries that did not require DNS resolution. There were no log messages from filterdns until it was killed and restarted, at which point it was functioning as expected.</p>
<p>The <code>filterdns</code> process was idle and not doing anything. Running <code>truss -fp</code> on its PID showed no activity.</p>
<p>When it was stuck:<br /><pre>
root 91126 0.0 0.2 101592 8936 - Is 14:10 0:09.41 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1
</pre><br />After killing it manually and restarting:<br /><pre>
root 6915 0.0 0.2 33880 8060 - Ss 09:47 0:00.06 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 30 -c /var/etc/filterdns.conf -d 1
</pre></p>
<p>Checking it randomly after, the main process was always in the "sleep" state (<code>Ss</code>), I didn't see the main process in the idle state (<code>Is</code>) again, but if you show all threads (<code>ps uxHaww | grep filterdns</code>) the individual threads are in the idle state.</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=376642018-08-14T13:36:48ZAnonymous
<ul><li><strong>Assignee</strong> set to <i>Renato Botelho</i></li></ul> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380852018-08-27T13:47:30ZAnonymous
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>13</i></li></ul> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380862018-08-27T13:51:41Z→ luckman212luke.hamburg@gmail.com
<ul></ul><p>Would this affect IP alias lists getting "stuck" and not updating? I ran into this last week - editing an Alias, added a few IPs and everything saved and looked good in the GUI, but when checking the actual pf tables they still had the old IPs. Only a reboot fixed it.</p>
<p>Was just wondering if this was the same issue...</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380872018-08-27T13:54:46ZJim Pingle
<ul></ul><p>That is a possible side effect. You can kill the <code>filterdns</code> daemon and then trigger a filter reload from <strong>Status > Filter Reload</strong> to restart it. Much faster than a reboot.</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380882018-08-27T13:58:35Z→ luckman212luke.hamburg@gmail.com
<ul></ul><p>Hmm. In that case, how about we add some tests to the alias edit php functions to query pfctl directly after a save and if it finds that the tables are not updating, then kill and restart the daemon automatically?</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380892018-08-27T14:00:11ZJim Pingle
<ul></ul><p>That's a rather ugly hack/kludge and it would be better to find out why it's failing and fix it directly.</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380902018-08-27T14:17:25Z→ luckman212luke.hamburg@gmail.com
<ul></ul><p>Agree it's less than ideal. But until a proper fix can be found a kludge might be safer than nothing, since aliases not updating could leave large security holes open.</p>
<p>I'm not sure where the source code for filterdns.c even is. It looks like it <em>used</em> to be in pfsense-tools but that's been archived and made private. Where to even look?</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380912018-08-27T14:30:30ZJim Pingle
<ul></ul><p>It is at <a class="external" href="https://github.com/pfsense/FreeBSD-ports/blob/devel/net/filterdns/files/filterdns.c">https://github.com/pfsense/FreeBSD-ports/blob/devel/net/filterdns/files/filterdns.c</a> but Renato is already looking at this currently so it will be fixed properly in this release.</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380922018-08-27T14:37:30Z→ luckman212luke.hamburg@gmail.com
<ul></ul><p>Sounds good!</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=380972018-08-28T06:12:28ZRenato Botelhorenato@netgate.com
<ul></ul><p>Jim Pingle wrote:</p>
<blockquote>
<p>I had this happen on my edge firewall which is running a snapshot from earlier this week. The pf tables with entries managed by <code>filterdns</code> were empty when it failed, even entries that did not require DNS resolution. There were no log messages from filterdns until it was killed and restarted, at which point it was functioning as expected.</p>
<p>The <code>filterdns</code> process was idle and not doing anything. Running <code>truss -fp</code> on its PID showed no activity.</p>
<p>When it was stuck:<br />[...]<br />After killing it manually and restarting:<br />[...]</p>
<p>Checking it randomly after, the main process was always in the "sleep" state (<code>Ss</code>), I didn't see the main process in the idle state (<code>Is</code>) again, but if you show all threads (<code>ps uxHaww | grep filterdns</code>) the individual threads are in the idle state.</p>
</blockquote>
<p>The I state indicates it's sleeping for over 20 seconds and per-se is not the problem because filterdns threads sleep for 1 minute so it will stay as S in the first 20 seconds and then move to I</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=381052018-08-28T07:43:53ZRenato Botelhorenato@netgate.com
<ul><li><strong>Status</strong> changed from <i>13</i> to <i>In Progress</i></li></ul> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=382422018-09-04T09:10:29ZAnonymous
<ul><li><strong>Assignee</strong> deleted (<del><i>Renato Botelho</i></del>)</li><li><strong>Target version</strong> changed from <i>2.4.4</i> to <i>48</i></li></ul><p>Currently unable to reproduce</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=382492018-09-04T13:09:23Z→ luckman212luke.hamburg@gmail.com
<ul></ul><p>I have definitely hit this one - yes it is hard to reproduce. But, if I hit it again, is it worth sending any sort of truss output or anything else to help track it down?</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=382512018-09-04T13:47:39ZRenato Botelhorenato@netgate.com
<ul></ul><p>Luke Hamburg wrote:</p>
<blockquote>
<p>I have definitely hit this one - yes it is hard to reproduce. But, if I hit it again, is it worth sending any sort of truss output or anything else to help track it down?</p>
</blockquote>
<p>Can you patch your local system the places that start filterdns and replace '-d 1' by '-d 10' to increase debug level? then if it fails it would be good to check the logs from /var/log/resolver.log</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=382712018-09-06T15:00:26Z→ luckman212luke.hamburg@gmail.com
<ul></ul><p>Ok, I've done that. For anyone else who wants an easy way, I made a patch that you can add via System Patches:<br /><a class="external" href="https://github.com/luckman212/pfsense/commit/72834bf677bdbd1cf78f6772b79abe4b3eaa8235">https://github.com/luckman212/pfsense/commit/72834bf677bdbd1cf78f6772b79abe4b3eaa8235</a></p>
<p>Don't forget to kill & restart filterdns after applying, or simply reboot.<br />To follow the resolver logs in realtime, you can use:<br /><pre>
clog -f /var/log/resolver.log
</pre></p>
<p>Also, I don't see it linked here so I want to add, this seems very related to <a class="external" href="https://redmine.pfsense.org/issues/8001">https://redmine.pfsense.org/issues/8001</a> which was reported on the forums here: <a class="external" href="https://forum.netgate.com/topic/124467/filterdns-stops-working">https://forum.netgate.com/topic/124467/filterdns-stops-working</a></p>
<p>Might literally be the same issue.</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=390562018-10-24T01:31:53Zkhaled osama
<ul><li><strong>File</strong> <a href="/attachments/2639">resolver.txt</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2639/resolver.txt">resolver.txt</a> added</li></ul><p>Dear All</p>
<p>i am affected with same problem<br />it happens every day approx.<br />i must kill filterdns service and restart to make worked again<br />i increased the debug level for filterdns<br />and i attached the resolver log file<br />best regards,</p>
<p>i linked the same post from netgate forum<br /><a class="external" href="https://forum.netgate.com/topic/124467/filterdns-stops-working/36">https://forum.netgate.com/topic/124467/filterdns-stops-working/36</a></p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=391822018-11-08T12:21:27ZLuiz Souzaluiz@netgate.com
<ul><li><strong>Assignee</strong> set to <i>Luiz Souza</i></li></ul> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=391922018-11-09T07:25:22ZLuiz Souzaluiz@netgate.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>This issue was one of fixes included in the new filterdns (version 2.0).</p>
<p>If you still have issues, please let us know.</p>
<p>Thanks!</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=391932018-11-09T07:25:33ZLuiz Souzaluiz@netgate.com
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=391982018-11-09T07:57:29ZRenato Botelhorenato@netgate.com
<ul><li><strong>Target version</strong> changed from <i>48</i> to <i>2.4.4-p1</i></li></ul> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=393352018-11-21T11:19:07ZLuiz Souzaluiz@netgate.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Closed</i></li></ul> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=398562019-01-24T06:24:18ZRudolf Mayerhofer
<ul></ul><p>Just ran into this on 2.4.4-p2 with a not updating alias table:</p>
<p>[2.4.4-RELEASE][root@fw2]/root: ps aux | grep filterdns<br />root 96457 0.0 0.0 49776 9336 - Is 10Jan19 0:10.67 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1<br />root 2975 0.0 0.0 6564 2468 0 S+ 12:54 0:00.00 grep filterdns</p>
<p>After killall -9 filterdns and applying changes to an alias filterdns started working properly again:</p>
<p>[2.4.4-RELEASE][root@fw2]/root: ps aux | grep filterdns<br />root 36300 67.0 0.0 47984 8016 - Ss 12:54 0:00.01 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1<br />root 39464 0.0 0.0 6564 2468 0 S+ 12:54 0:00.00 grep filterdns</p>
<p>Unfortunately, at this point in time I do not have any more debug output available. I will continue to monitor this and add more information as it will be available to me.</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=399802019-02-18T07:21:11ZRobert Gijsen
<ul></ul><p>2.4.4-RELEASE-p2, I've had this multiple times. At the moment I can even sort of reproduce it.<br />When adding hosts to an alias my AD DNS server logs:</p>
<p>2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Rcv <pfsense IP> a463 Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)</p>
<p>2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Rcv <pfsense IP> 519a Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)</p>
<p>2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Snd <pfsense IP> a463 R Q [8081 DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)</p>
<p>2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Snd <pfsense IP> 519a R Q [8085 A DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)</p>
<p>This is an external host, i.e. a DNS that needs to be externaly resolved by our DNS servers. That seems to work fine result gets send back to pfSense. However the host does NOT end up in the table for that alias. When I add another DNS, same domain, so hosted at the same DNS on internet, that works fine. I tried others like <a class="external" href="http://www.tweakers.net">www.tweakers.net</a>, <a class="external" href="http://www.nos.nl">www.nos.nl</a> or bbc.co.uk I have the same success loggings in my DNS debug log, and they DO end up in the alias table as well. At first I though the issue was with hosts that are already in a table somewhere, but that doesn't seem to be the case. Most internal names I tried now don't end up in that table either.</p>
<p>pfSense Resolver log:<br />Feb 18 12:47:14 filterdns Adding host <Host that gets added to the alias> (I just added that one in the alias)<br />Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <Host that gets added to the alias><br />Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <host that does NOT end up in table> (I just added that one in the alias as well)<br />Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <a class="external" href="http://www.ict-net.nl">www.ict-net.nl</a></p>
<p>The host that does NOT end up in table here, is by the way successfully added to some other aliasses, where it works just as expected. But for this alias I am missing the 'Adding host' in the pfSense log.</p>
<p>I tried creating a new alias, with the same three hosts as in the alias I used above. Here NONE of them end up in the table, after waiting for about 20 minutes, while in the alias used above two out of three (and the same two every time, no matter what order I put them in) work. Then I added <a class="external" href="http://www.tweakers.net">www.tweakers.net</a> as another try, and that one gets in there immediately.<br />I again killed filterdns, restarted it and poof - the tables immediately got filled as they should. So it seems filterdns is partially functional - some hosts get added, some aren't.</p>
<p>Tell me what loggings you need. As it seems I can now reproduce this at will (also on my second carp / HA node by the way) I can probably give all needed logs.</p> pfSense - Bug #8758: filterdns stops working on a regular basis.https://redmine.pfsense.org/issues/8758?journal_id=399812019-02-18T07:25:08ZJim Pingle
<ul></ul><p>This is a closed/resolved issue. If you have problems with filterdns, they are likely already covered by <a class="issue tracker-1 status-3 priority-4 priority-default closed" title="Bug: Alias content is sometimes incomplete when an alias contains both FQDN and IP address entries (Resolved)" href="https://redmine.pfsense.org/issues/9296">#9296</a> -- add your notes there.</p>