pfSense

https://forum.netgate.com/topic/160005/pfsense-2-50-snapshots-have-been-dying-for-the-past-couple-of-days

I have the same problem. it happens only when the option "Register DHCP leases in the DNS Resolver" is set.

it looks like dhcp tries to restart the unbound service because of a new dhcp lease but unbound does not start again.

Jan 29 13:45:07 heimdall1 dhcpleases¹⁵⁰⁶⁶: Could not deliver signal HUP to process 33623: No such process.
Jan 29 13:45:05 heimdall1 dhcpleases¹⁵⁰⁶⁶: Could not deliver signal HUP to process 33623: No such process.
Jan 29 13:45:04 heimdall1 dhcpleases¹⁵⁰⁶⁶: Could not deliver signal HUP to process 33623: No such process.
Jan 29 13:45:02 heimdall1 kernel: pid 33623 (unbound), jid 0, uid 59: exited on signal 11

Keep the discussion on the forum. If it's still happening, there is no evidence there. Last post was over a week ago and that last response implies their problem was resolved. No indication it's ongoing on current snapshots.

In the "competitor's" forum, there are several pages of error descriptions and error analyses for Unbound 1.13.0. Also a bug report is available at freebsd.
Ignoring is a great bug fix.

PS: I can confirm Daniel's observation. It seems to be related to the option "Register DHCP leases in the DNS Resolver".

Behavior on other systems (even FreeBSD) isn't directly relevant to pfSense software. They may be similar, but it's not 100% the same. We need more information about how it happens.

I run my edge, and several others, with DHCP registration on and have zero problems on snapshots with the DNS Resolver. If there is a problem, there is much more to it. Keep discussing it on the forum for now.

If we can't reproduce it, we have no way to debug it or test if it's fixed.

Furthermore, the fix committed to FreeBSD is already present on the latest snapshots and has been for the last three weeks:

https://github.com/pfsense/FreeBSD-ports/commit/9510cfe4c453c2f589b2b065b9f42a85e7f3c5ba

If it's still happening, it may not be the same root cause, which still means we need more information. On the forum thread. Not here.

1.13.1 cherry-picked to 2.5.0 branch

I never had any problem with the core system of pfSense on production. Today my unbound died. (about 5 hours after upgrading to 2.5.0-Release). Just before it happened some new Wifi client registered.
I disabled "Register DHCP leases in the DNS Resolver" and "Register DHCP static mappings in the DNS Resolver" (those where both enabled on my setup).
Ill report back if it happens again.

Pim Janssen wrote:

I never had any problem with the core system of pfSense on production. Today my unbound died. (about 5 hours after upgrading to 2.5.0-Release). Just before it happened some new Wifi client registered.
I disabled "Register DHCP leases in the DNS Resolver" and "Register DHCP static mappings in the DNS Resolver" (those where both enabled on my setup).
Ill report back if it happens again.

You will need to wait until version 1.13.1 is available and installed on your system to make sure your test is valid.

The forum thread linked above has instructions for installing the updated version manually from the snapshot repository:
https://forum.netgate.com/topic/160005/unbound-crashes-periodically-with-signal-11/57

Eventually we'll also have a build available in the 21.02/2.5.x repository for manual upgrades.

This is now in the 2.5.0 repository. To upgrade manually, run the following from an ssh or console shell prompt (not the GUI):

pkg upgrade -fy unbound; pfSsh.php playback svc restart unbound

Will the update be made available to 21.02 soon? My 2.5.0 box finds it, but my 21.02 box does not.

Thanks!

On 21.02, in the meantime, the following will upgrade unbound:

pkg add -f https://files01.netgate.com/pfSense_v2_5_0_amd64-pfSense_v2_5_0/All/unbound-1.13.1.txz; pfSsh.php playback svc restart unbound

No need for that now, it's live in the 21.02 repository now that 21.02-p1 has been released to address SG-3100 stability.

It's not a part of 21.02-p1, but it's in the pkg repository, so you might happen to pick it up as a part of the upgrade like any other available package update post-release.

I was seeing unbound simply die about once a day since upgrading to 2.5.0-RELEASE. No info as to why in the service's logs available via the web UI, but by running (dmesg | grep bound) I found it was due to segmentation violations (i.e., pid 65353 (unbound), jid 0, uid 59: exited on signal 11). I therefore ran (pkg upgrade -fy unbound) yesterday morning.

The service is not dying with SEGV any longer, at least not yet, but it is restarting regularly. Not having monitored it before, I don't know if that is expected. I suspect not, so I'm reporting it.

Feb 25 14:43:34     unbound     4750     [4750:0] info: start of service (unbound 1.13.1).
Feb 25 14:43:34     unbound     4750     [4750:0] notice: init module 0: iterator
Feb 25 14:43:34     unbound     4750     [4750:0] notice: Restart of unbound 1.13.1.
Feb 25 14:43:34     unbound     4750     [4750:0] info: server stats for thread 3: requestlist max 0 avg 0 exceeded 0 jostled 0
Feb 25 14:43:34     unbound     4750     [4750:0] info: server stats for thread 3: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Feb 25 14:43:34     unbound     4750     [4750:0] info: server stats for thread 2: requestlist max 0 avg 0 exceeded 0 jostled 0
Feb 25 14:43:34     unbound     4750     [4750:0] info: server stats for thread 2: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Feb 25 14:43:34     unbound     4750     [4750:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
Feb 25 14:43:34     unbound     4750     [4750:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Feb 25 14:43:34     unbound     4750     [4750:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
Feb 25 14:43:34     unbound     4750     [4750:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Feb 25 14:43:34     unbound     4750     [4750:0] info: service stopped (unbound 1.13.1).
Feb 25 14:43:34     unbound     4750     [4750:0] info: start of service (unbound 1.13.1).
Feb 25 14:43:34     unbound     4750     [4750:0] notice: init module 0: iterator
Feb 25 14:43:34     unbound     4750     [4750:0] notice: Restart of unbound 1.13.1.
Feb 25 14:43:34     unbound     4750     [4750:0] info: 0.262144 0.524288 4
Feb 25 14:43:34     unbound     4750     [4750:0] info: 0.131072 0.262144 8
Feb 25 14:43:34     unbound     4750     [4750:0] info: 0.065536 0.131072 19

Registered just to add to this as DNS is quite important part of the network and needs to be fixed.
I am too having problems with unbound randomly just stopping and not starting after upgrade to 2.5.0
once it happened with older version of unbound that was pushed with 2.5.0
then I manually updated unbound using suggested method
pkg upgrade -fy unbound; pfSsh.php playback svc restart unbound

but today I have it crash again
also in system logs I see it restarting at some intervals.

Feb 26 09:54:33 unbound 51482 [51482:0] info: start of service (unbound 1.13.1).
Feb 26 09:54:33 unbound 51482 [51482:0] notice: init module 1: iterator
Feb 26 09:54:33 unbound 51482 [51482:0] notice: init module 0: validator
Feb 26 09:54:33 unbound 51482 [51482:0] notice: Restart of unbound 1.13.1.

and so on...

until next time it will die and not restart.

I am running dns resolver with
DHCP Registration
and
Static DHCP
settings enabled
otherwise
default settings.

It is normal for Unbound to restart often when DHCP hostname registration is on. This bug is only for the actual crash (segfault), not other behavior.

Having segfault crashes on 1.13.1:

https://forum.netgate.com/topic/161372/2-5-0-unbound-1-13-1-exited-on-signal-8-sigfpe-floating-point-exception/3?_=1614303643873

I have the same issue, after updating two of my pfsense boxes I see abut 4 to 5 messages from each per hour

"Service Watchdog detected service unbound stopped. Restarting unbound (DNS Resolver)"

I'm losing DNS every day or so with pfsense 2.5. I'm using the latest from "pkg update".
If there's anything I can do to help - logs, try a special version, etc. I'm happy to help.
I'm running 2.5 on a non-critical firewall, so I'm willing to try things to get rid of this bug.

Assuming this is the same segfault others are hitting with Unbound they are still investigating it upstream: https://github.com/NLnetLabs/unbound/issues/411

There is a patch to have Unbound log more detailed debugging information if it does crash in the place mentioned in that issue:
https://github.com/NLnetLabs/unbound/commit/269c168f7e58dc3a18ff0148fd8cce959f71bad7

We can look into adding that to snapshots to at least help gather more information for upstream.

Christian Borchert wrote:

Here's the logs from a second signal 11 crash a few hours later

Mar 10 03:44:09 router kernel: pid 87756 (unbound), jid 0, uid 59: exited on signal 11

There isn't anything relevant in those. You can keep these on the forum if you like, but they aren't going to be useful here and clutter up the issue.

Jim Pingle wrote:

Updating subject for release notes.

If Unbound doesn't find/fix the issue in 1.13.1 soon we may consider rolling Unbound back to 1.12.0 if it's viable. The only CVE addressed since then isn't a major concern on pfSense, and though some of the bug fixes in 1.13.x are beneficial, the instability is a bigger problem.

Updating manually to 1.13.1 did not solve issue completely it may have decreased crashes to 1-4 times per week but it's still crashing, downgrade would be acceptable solution at least in my opinion as I never had problems with unbound crashing in previous versions of pfsense.

I hope the decision is not made to roll back unbound, as its just going back to old code, when the better decision might be to disable register dhcp leases by default, I am not sure of the benefits of that option for most people's configuration.

I think as long as you give people a heads up and the option remains for the few that need the option to enable it again I think it would be accepted by the community, everyone on the forum who I advised to disable the option reported their issues been fixed.

Chris Collins wrote:

I hope the decision is not made to roll back unbound, as its just going back to old code, when the better decision might be to disable register dhcp leases by default, I am not sure of the benefits of that option for most people's configuration.

Sure Unbound 1.12.0 is "old code" but if you look at the Changelog, there isn't anything life changing from 1.12.0 to 1.13.x that would hurt us significantly. I'd rather roll back to code we know is stable than disable a feature used by tens of thousands of users because a handful of people still encounter instability that may (or may not) be related to that feature.

Chris Collins wrote:

I hope the decision is not made to roll back unbound, as its just going back to old code, when the better decision might be to disable register dhcp leases by default, I am not sure of the benefits of that option for most people's configuration.

I think as long as you give people a heads up and the option remains for the few that need the option to enable it again I think it would be accepted by the community, everyone on the forum who I advised to disable the option reported their issues been fixed.

Maybe in your case it is not important but for my it is I would gladly downgrade to unbound 1.12.0 if it solves my problem (is there a supported way to manually do it?)
I rely on hostnames for my internal network management (static ip hosts and dynamic ip hosts) heavily and disabling registering dns names for hosts would break my workflows.
I even consider moving to separate DNS/DHCP servers. for now watchguard rebooting unbound when it's crashed is temporary fix for me. I set up mail notifications for watchgourd rebooting unbound and getting email almost every single day that unbound has crashed.

Vaidotas, static DHCP should probably be used if you rely on hostnames so much. The feature in general has been the cause of so many problems, I have for several years been seeing reports of problems caused by it.

However I didnt propose removing the feature, just toggling the default.

Can confirm the same happening on my system. Unbound crashed with an interval of one week and always at night. And it only happened twice since I updated to 2.5.0 a long time ago.

Apr 1 01:02:14 pfSense kernel: pid 42679 (unbound), jid 0, uid 59: exited on signal 11
Apr 8 07:03:27 pfSense kernel: pid 43848 (unbound), jid 0, uid 59: exited on signal 11

Looking at it, it seems to be aligned with the pfBlockerNG update schedule although not sure if this is not a coincidence. Is there anything else I could provide to make troubleshooting easier? Some more logs?

Update: Just did some digging in the logs so see what happened just after and before the crash.

Apr 1 01:00:00 pfSense php⁵⁶⁹⁸: [pfBlockerNG] Starting cron process.
Apr 1 01:01:00 pfSense php⁶⁴⁶³⁸: rc.dyndns.update: phpDynDNS (vpn.szymon.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Apr 1 01:02:14 pfSense kernel: pid 42679 (unbound), jid 0, uid 59: exited on signal 11
Apr 1 01:02:32 pfSense php⁵⁶⁹⁸: [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload

Apr 8 07:00:00 pfSense php¹⁴²⁶¹: [pfBlockerNG] Starting cron process.
Apr 8 07:02:35 pfSense php¹⁴²⁶¹: [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Apr 8 07:02:35 pfSense php¹⁴²⁶¹:
Apr 8 07:03:27 pfSense kernel: pid 43848 (unbound), jid 0, uid 59: exited on signal 11

I'm experiencing this issue as well.

It seems to be preceded by unbound going 100% cpu for several minutes, during which 'unbound-control stop' will time-out, the stop and restart buttons in the webui just spin for 30 second then return without doing anything, and even doing a kill on the unbound process does nothing. DNS queries will still go through for a while but it seems to eventually dig itself a hole it can't escape, after which it will stop processing queries. The only thing I've been able to do is reboot the entire box (XG-7100U), which is... not ideal.

Watchdog-ing unbound doesn't seem to help, as the process never seems to exit, it just stops responding.

Is there any hope for a fix? It's been quite a while and I don't see any progress.
I'm happy to try to help, but not sure what to do. I'm thinking of moving to another DNS approach.
Thanks.

This really is a show stopper for us, I've upgraded from 2.4.5 2 days ago and now our unbound server is crashing constantly!

And the proposed workaround is not really an option for us as we rely on this feature to manage our workstations remotely...

We're reverting to unbound 1.12 in order to restore stability. We have to backport at least one CVE for it, so it's taking extra time before we can put out an update. Preliminary testing looks like the problem is improved.

Is there a release of the reverted unbound to try? I'm willing to try it.
I'm now crashing 3 or 4 times a day.

Had crash that even watchdog did not recover. tried manually restarting unbound via web GUI, status was showing as ok but could not resolve any host names, but IP addresses were no problem. Had to reboot device. Thank you for looking into it as it is starting to drive me crazy.

Snapshots for 2.6.0 and 21.05 have had Unbound 1.12.x for a couple weeks now.

I created #11915 to note that since that is a temporary measure and this root issue is not yet solved.

Jim Pingle wrote:

Snapshots for 2.6.0 and 21.05 have had Unbound 1.12.x for a couple weeks now.

I created #11915 to note that since that is a temporary measure and this root issue is not yet solved.

Is there a way to install the reverted Unbound in 2.5.1?

Mike Farmwald wrote:

Jim Pingle wrote:

Snapshots for 2.6.0 and 21.05 have had Unbound 1.12.x for a couple weeks now.

I created #11915 to note that since that is a temporary measure and this root issue is not yet solved.

Is there a way to install the reverted Unbound in 2.5.1?

The unbound112 build is available in the pkg repository but we're still working on a good set of pkg commands to swap it out safely. Since it's such a critical core dependency it's not quite as simple as removing one and adding the other. But feel free to try it out (hopefully safely in a lab first).

Jim Pingle wrote:

The unbound112 build is available in the pkg repository but we're still working on a good set of pkg commands to swap it out safely. Since it's such a critical core dependency it's not quite as simple as removing one and adding the other. But feel free to try it out (hopefully safely in a lab first).

Is the "good set of pkg commands" available somewhere already ? I couldn't find it, nor find any information on how to swap unbound for unbound112 safely.
If not, do you know where such procedure will be made available ?

I can see that pfSense 21.05 was released with unbound 1.12 instead of 1.13, but nothing for pfSense CE yet.

As an ugly workaround, I'm using "Service Watchdog" package to restart unbound when it crashes. This happens every few minutes but seems to work alright.

I'm on CE 2.5.1 with unbound 1.13.1

For those commenting about restarting the service - that didn't help me. Had to restart the firewalls.

This 5 month old issue with no resolution and no suggested workaround should be getting a lot more exposure such as on the Netgate Blog because it's a dangerous landmine that can strike at any time and you have zero warning or clue when it hits you.

Some weeks after updating to 2.5.1 it suddently caused all our site's services to become unusably slow and luckily I thought to ping some servers via LAN IP's and noticed that was working fine which made me guess it was some kind of DNS issue, and maybe restarting the service would help (it didn't) and then went to the old fallback of rebooting the firewalls which fixed the issue (until the next time). You have to be lucky and thinking pretty clearly and calmly to figure this out while all of your stuff isn't working and alerts and emails are coming in.

The only "good" part is this happened at 9pm on a Sunday and not 3am on a Monday. Things can always be worse.

This problem is very much interfering with my network operations. I have watchdog restarting unbound but
it can take several seconds or more before I can resolve again. This is not a small problem.
Please provide some status update.

Many Thanks.

There is no change since the last updates already covered above in previous comments:

• Unbound still hasn't put out a fixed release
• We added the latest relevant commits from the Unbound project to Unbound 1.13.1 on CE 2.6.0 and Plus 21.09 snapshots for testing
• We moved back to Unbound 1.12.x for Plus 21.05 and for CE 2.5.2 which is in RC stage now, and will be coming out soon. See #11915

The workaround is the same as it has been, disable DHCP lease registration in the DNS resolver or try a development snapshot.

Jim Pingle wrote:

There is no change since the last updates already covered above in previous comments:

• Unbound still hasn't put out a fixed release
• We added the latest relevant commits from the Unbound project to Unbound 1.13.1 on CE 2.6.0 and Plus 21.09 snapshots for testing
• We moved back to Unbound 1.12.x for Plus 21.05 and for CE 2.5.2 which is in RC stage now, and will be coming out soon. See #11915

The workaround is the same as it has been, disable DHCP lease registration in the DNS resolver or try a development snapshot.

Hi Jim,
I updated to CE 2.6.xx and the unbound failures seem to be fewer but still a problem.

Jul 1 09:02:56 unbound 80302 [80302:0] info: 512.000000 1024.000000 6
Jul 1 09:02:56 unbound 80302 [80302:0] notice: Restart of unbound 1.13.1.
Jul 1 09:02:56 unbound 80302 [80302:0] notice: init module 0: validator
Jul 1 09:02:56 unbound 80302 [80302:0] notice: init module 1: iterator
Jul 1 09:02:56 unbound 80302 [80302:0] info: start of service (unbound 1.13.1).
Jul 1 09:02:56 unbound 80302 [80302:0] info: service stopped (unbound 1.13.1).
Jul 1 09:02:56 unbound 80302 [80302:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Jul 1 09:02:56 unbound 80302 [80302:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
Jul 1 09:02:56 unbound 80302 [80302:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Jul 1 09:02:56 unbound 80302 [80302:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
Jul 1 09:02:56 unbound 80302 [80302:0] notice: Restart of unbound 1.13.1.
Jul 1 09:02:57 unbound 80302 [80302:0] notice: init module 0: validator
Jul 1 09:02:57 unbound 80302 [80302:0] notice: init module 1: iterator
Jul 1 09:02:57 unbound 80302 [80302:0] info: start of service (unbound 1.13.1).
Jul 1 09:02:57 unbound 80302 [80302:0] info: generate keytag query _ta-4f66. NULL IN
Jul 1 09:02:58 unbound 80302 [80302:1] info: generate keytag query _ta-4f66. NULL IN

I look forward to the unbound devs resolving this problem. Having to use an ip address just to get to the router to restart things when even watchdog fails
is disheartening.

Cheers

Remo Wylliams wrote:

I updated to CE 2.6.xx and the unbound failures seem to be fewer but still a problem.

Are you still seeing a crash/core dump from unbound? Check the main system log. The resolver log only shows it restarting (which is normal when it updates data from DHCP leases), that log snippet doesn't show the crash.

No I'm not seeing any crashes on unbound. Didn't know about the DHCP leases.
Thanks again.

I take that back. no sooner do I say it's working then.. BANG..
No crash in the log but completely hung DNS and restarting resolver causes unbound to fail.
Jul 1 13:34:09 php-fpm 1720 /status_services.php: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1625164449] unbound[43151:0] error: bind: address already in use [1625164449] unbound[43151:0] fatal error: could not open ports'

so it was stuck and wouldn't respond to the signal.
I had to reboot my router to post this.
So less crashy more stubborn.

Martin Müller wrote:

"Unbound 1.13.1 was just released a few days ago

I had the same problem with pfsense 2.5.1 running unbound version 1.13.1 that randomly crashes.
I upgraded to pfsense 2.5.2 which runs unbound 1.12.0 and that fixes the issue ;-)

The most recent 2.5.2 version of pfsense seems to work great. or at least in a way I haven't found a problem.
Thanks for the update.

I have been running 21.05 for over a month and haven't seen any unbound crashes at all on 1.12. We will want to retest if we move up to a newer version of unbound but this version appears stable for now.

Kris Phillips wrote in #note-61:

I have been running 21.05 for over a month and haven't seen any unbound crashes at all on 1.12. We will want to retest if we move up to a newer version of unbound but this version appears stable for now.

Plus 21.09 and CE 2.6.0 snapshots have a patched version of Unbound 1.13.x for testing. If anyone can reliably reproduce the crash, that's the next thing to test.

I've imported more patches from upstream and bumped unbound to 1.13.1_3. This version will be available with next round of snapshots and must be tested to make sure it fixes all issues

That is odd, the log also didn't show a crash, but a clean stop and start. That is also a very old log, maybe you uploaded the start of the log instead of the end? It appears to be in reverse order which is difficult to follow.

If it was a similar crash to the previous issues, you would see the crash in the main system log. You can filter that for messages containing unbound to narrow it down. You'd see messages like those in note 3, 18, and so on above.

Jim Pingle wrote in #note-67:

If it was a similar crash to the previous issues, you would see the crash in the main system log. You can filter that for messages containing unbound to narrow it down. You'd see messages like those in note 3, 18, and so on above.

Thanks. Turns I did have crash messages, but in syslog, not unbound log.


Aug 11 14:53:49 router kernel: pid 47761 (unbound), jid 0, uid 59, was killed: out of swap space
Aug 11 14:53:50 router kernel: pid 17663 (pcscd), jid 0, uid 0, was killed: out of swap space


That could be unrelated
(Though I have no swap (it's on flash) and memory shows 41% used. No idea what that's about)

OK that is unrelated to this bug. It is #12095 which can be mitigated by the patch on #11933.

I've tested this a bit in 1.13.2 on 22.01 and have been unable to reproduce DHCP/DNS crashes with the latest unbound running 1.13.2. We'll want more expansive testing, but in a few hours of trying to make it crash with repeated DHCP lease requests and DNS registrations I don't see any issues.

There are two problems here.

1. There is no watchdog restarting the unbound service
2. An invalid unbound configuration is being written by dhcpleases to /var/unbound/

Jan 17 12:00:04 unbound[98083]: [98083:0] notice: Restart of unbound 1.12.0.                                                                                                        
Jan 17 12:00:04 unbound[98083]: [98083:0] fatal error: Could not read config file: /unbound.conf. Maybe try unbound -dd, it stays on the commandline to see more errors, or unbound-
checkconf                                                                                                        

Notice that dhcpleases HUP's unbound several times in a row. It's very likely that `dhcpleases` is doing non-atomic writes to /var/unbound/dhcpleases_entries.conf, and unbound happens to read the file mid-write.

PfSense: 2.5.2-RELEASE (amd64)
Unbound: 1.12.0

Jim Pingle, this is still an open issue.

I hate to bring up a sore point especially in a closed ticket, but this is still happening for me on two up-to-date pfSense 2.6.0 / 22.01 installations. I upgraded from 2.5.2 / 21.05.2 and was very hopeful, but while 22.01 / 2.6.0 may have made the issue occur less frequently, it did not fix the underlying problem.

I have a NAGIOS instance which monitors this and I get a notification whenever nslookups of statically defined DHCP entries results in a NXDOMAIN. I have written a quick script which disables DHCP, deletes all the current DHCP reservations, restarts DHCP then restarts unbound on both installations, and the problem goes away for a few days, but it does come back.

For my pfSense instances, I have noticed that it seems to be with resources that have static DHCP entries that are currently offline. I have a laptop with a statically defined DHCP entry, and it seems the DNS failure occurs when my laptop is NOT connected to the network and its lease is shown as 'offline'. To be clear: NORMALLY, I can nslookup 'laptop.example.com' and get its statically defined IP address whether or not the laptop is online. In the failure mode, I get an NXDOMAIN for the same lookup whether or not my laptop is online. However, I have never seen the failure mode START when my laptop is online, only when it has been off for a period of time, and the lease is shown as 'offline'.

In any case, I do not want to be that person who only complains, so please let me know what I need to do to help further.

Why is this closed ??
All was ok for my pfsense until a power outage.
I have pfsense 2.6 up to date and it has been running for months without any problem.
After restart I get this again and again.
So I desactivated lease registration but, what can I do to enable it back ?

This is great news! Thanks Christian McDonald for diving into this issue. Even when it was almost a complete rewrite of the MaxMindDB Python Module and outside of the base pfSense code! It's very much appreciated!

Will be rolling back the safety belts that was added to pfBlockerNG to prevent unbound python mode from crashing which will allow for users to re-enable the dhcp leases option for the pfSense versions that have these recent changes.