pfSense

This has happened with a notification that acb.netgate.com couldn't be resolved. Issue was fixed by rebooting the OS. I did look into /var/db/notifyqueue.messages and saw it still has data in it:

a:1:{s:5:"mails";a:1:{s:4:"item";a:2:{i:0;a:3:{s:4:"time";i:1677139260;s:4:"type";s:4:"mail";s:3:"msg";s:118:"The following CA/Certificate entries are expiring:
Certificate: kanako-test (638dc1866bf9d): Expiring soon, in 10 days";}i:1;a:3:{s:4:"time";i:1677225660;s:4:"type";s:4:"mail";s:3:"msg";s:117:"The following CA/Certificate entries are expiring:
Certificate: kanako-test (638dc1866bf9d): Expiring soon, in 9 days";}}}}

It doesn't appear related to the content of the file, I copied that into mine and triggered a notification and it sent that just fine and wiped the queue after. There must be some other component to triggering it. I don't see anything obvious in the code that looks like a problem except for maybe the locking as I mentioned on your forum thread

Anyone who has seen this happen, please install the System Patches package and then create an entry for 053f60e56d09ba711af245c7f0ce6a06673ccc89 and apply that change, then try to trigger notifications as before to see if the loop happens again. If that does fix it, then this can be combined with #14031

I applied the system package but that did not solve the issue; its still sending e-mails out constantly and not clearing /var/db/notifyqueue.messages

Do you get an error if you try to erase /var/db/notifyqueue.messages by hand? You can try rm /var/db/notifyqueue.messages from a shell prompt.

While the current code should be OK, you can also try this change which is a slightly different approach to ensuring the message queue array gets reset.

diff --git a/src/etc/inc/notices.inc b/src/etc/inc/notices.inc
index 539698b821..aef4fdd8bd 100644
--- a/src/etc/inc/notices.inc
+++ b/src/etc/inc/notices.inc
@@ -253,7 +253,10 @@ function notices_sendqueue() {
                if (file_exists("{$g['vardb_path']}/notifyqueue.messages")) {
                        $messages = unserialize(file_get_contents("{$g['vardb_path']}/notifyqueue.messages"));
                        $messagequeue = $messages;
-                       $messages['mails']['item'] = array(); // clear all items to be send
+
+                       /* Reset message queue */
+                       $messages = [];
+                       array_set_path($messages, 'mails/item', []);
                        file_put_contents("{$g['vardb_path']}/notifyqueue.messages", serialize($messages));
                        unset($messages);
                }

I deleted the content of the file and let some notifications go through. That appears to have solved the issue; so I'm wondering if the previous content for some reason wasn't being properly emptied? Current content of the file is:

a:1:{s:5:"mails";a:1:{s:4:"item";a:0:{}}}

Curious though I can't see a reason why it would let you do that by hand but the code would fail. For now I'd apply that last diff I posted and keep letting notifications run to see if it happens again.

I finally had a system in my lab get stuck doing this, it was a fresh image of a 23.05 dev snapshot, restored a config with packages, and then it kept looping sending the same notification.

Looking at the process list, there was one instance of notify_monitor.php stuck that was somehow running as the nut user:

nut     93943   0.0  1.8  97080 36696  -  S    15:33    0:00.78 /usr/local/bin/php-cgi -q /usr/local/bin/notify_monitor.php

None of the notifications were from nut. There was only one notification that was triggered by pfSense itself:

15:33:13 Package reinstall process finished successfully

Killing that one process stopped the flood but the queue wasn't emptied.

Running /usr/local/bin/php-cgi -q /usr/local/bin/notify_monitor.php by hand, it takes a long time to finish, but also doesn't get stuck in a loop. After running it manually, the queue is cleared.

Not sure why that was triggered via nut but at least that's a lead.

I do have nut installed on my system. It hasn't repeated the process since my last report; maybe its a rare sequence of events that have to happen?

I can confirm I have this issue too and I have nut installed. Actually, I am suffering from it now and can't reboot as the Misses is watching tv ;-) , hence I am looking for a solution and found this bug report.

On my side it is always a Service Watchdog message because igmpproxy has crashed. Every time igmproxy fails this issue occurs. I am able to trigger this error by removing one of the network cables, so if I need to test something, please let me know.

Come to think of it: when I remove the network cable, the nut server loses connection with the UPS, so probably it throws an error too at that moment.

I was able to reproduce this on demand by triggering a notification from nut and a notification from the system shortly after:

require_once('service-utils.inc');
require_once('notices.inc');
restart_service('nut');
sleep(5);
file_notice('blah', 'Test message');

As I thought it was because the unprivileged user could not write the message queue, but it could read the message queue. So it would keep going in a loop thinking there were more messages to send.

I added some safety belt checks to hopefully prevent this from happening.

You can install the System Patches package and then create an entry for c5faa351c1ef6d4555478a7f50b3a16ece7e0b2a to apply the fix.

I can confirm the fix is working for me. I don't see any repeats anymore. Thanks Jim!

Tested on 23.01

I was able to reproduce this issue.
After applying c5faa351c1ef6d4555478a7f50b3a16ece7e0b2a, the issue was fixed.

Jim, I tested this patch and the end effect, at least for 23.01, is that non root notifications just silently fail.

Two things:

• Is there some other way that this can be addressed that still allows non root notifications to function? The way it currently sits, this would require all packages that use notifications to be run as root, which seems rather undesirable.

• When the notification does fail, it would be desirable if there were some form of error in the log.

That's not a problem we can solve here. Somehow the unprivileged process would have to submit a message into the queue and then some other privileged process would have to come along after and send it. Right now both are handled by the same user that triggered the notification. But that means we'd have to setup some other mechanism (daemon, cron script, etc) that would run the message queue and that's way out of scope for this issue. That would be a completely fresh feature request that would require designing a proper solution to implement in a future version.

Given that non-root package notifications worked previously, I think users are going to perceive this as a regression (bug).

Is there a recommendation as to how we should address the immediate issue? Should we change all packages to run as root?

They absolutely did not work correctly (hence this bug). They failed in new/different ways on 23.01 but they also failed on older versions. The times it happened to work were purely by chance.

The only package I'm aware of that is affected is nut, however that package wants to handle things is undecided. There are no defined best practices there currently, but this particular change is complete as-is.

Hmm... "the times it happened to work were purely by chance" seems quite a bit overstated. It seems to be rather the opposite--the number of times that notifications have failed appears to have been incredibly rare, as evidenced by the difficulty in reproducing this bug.

Regardless, do you have a recommendation for another approach to address the issue of email notifications from non root entities? Or do I need to change the NUT package to always run as root?

How to properly send notifications from nut is not a topic for this Redmine. Feel free to discuss it more on the forum.

Thanks. I submitted a PR that addresses the issue.

I would like to propose a small change to this fix. It is a "one liner," which maintains the ability to send notifications as a non root user. I think this is a reasonable approach that protects pfSense from the spamming problem while allowing package maintainers to be able to run processes without high privilege. I would appreciate it if you could consider this. Thanks.

--- notices.inc.old    2023-04-23 09:44:28.161806000 -0700
+++ notices.inc    2023-05-02 09:36:47.548761000 -0700
@@ -356,7 +356,7 @@

     /* Store last message sent to avoid spamming */
     @file_put_contents("/var/db/notices_lastmsg.txt", $message);
-    if (!$force) {
+    if ($force == false && posix_geteuid() == 0) {
         notify_via_queue_add($message, 'mail');
         $ret = true;
     } else {

It's too late for this issue / Plus 23.05, but if you want to open a new Redmine issue and propose it there we can consider it for 23.09/2.7.0.