pfSense

I decided to test the OPNsense igmp proxy 0.3 package by force installing it over the version (0.2.1) that comes with pfSense 2.5

like this: pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/21.1/latest/Latest/igmpproxy.txz

I did about 12 reboots with this version and no kernel panic so far. So, I guess igmp proxy is really the culprit behind those kernel panics (or the one that triggers something that leads to a kernel panic) and version 0.3 fixes that behaviour.

I do however still have an issue. pfSense does boot properly, starts everything but when it first starts IGMP proxy, it gives this error (on every bootup):

Jan 29 03:50:41 [myhostname] igmpproxy⁴⁰⁵⁸⁷: MC-Router API already in use; Errno(48): Address already in use

It seems like igmp proxy dies right after that error.

Not sure why this happens or what may conflict with it. Since I have installed service watchdog and configured it to monitor and restart igmpproxy in case of failure, igmpproxy gets restarted and then continues to work without any issue after that.

So, 0.3 seems to help or fix the kernel panic issue at least which is good so I think it would be a good idea to release it with 2.5 to prevent kernel panics at least. igmpproxy still quits on first startup however after initial boot but if you restart it manually (or by some other mean like service watchdog), it will restart without any issue or error and continue to work after that.

Confirmed, I see 0.3 now, thank you.

I still need to run watchdog to get the process restarted after initial bootup because of this error:

Feb  3 22:55:33 xxxxxx igmpproxy [70274]: MC-Router API already in use; Errno(48): Address already in use

It seems to happen when igmp proxy tries to start up here

    switch ( Err = enableMRouter() ) {
    case 0: break;
    case EADDRINUSE: my_log( LOG_ERR, EADDRINUSE, "MC-Router API already in use" ); break;
    default: my_log( LOG_ERR, Err, "MRT_INIT failed" );
    }

while doing this

int enableMRouter(void)
{
    int Va = 1;

    if ( (MRouterFD  = socket(AF_INET, SOCK_RAW, IPPROTO_IGMP)) < 0 )
        my_log( LOG_ERR, errno, "IGMP socket open" );

    if ( setsockopt( MRouterFD, IPPROTO_IP, MRT_INIT,
                     (void *)&Va, sizeof( Va ) ) )
        return errno;

    return 0;
}

For some reason, this seems to fail and triggers this error and it then quits, only on initial pfSense bootup sequence. It seems like there is something preventing igmp proxy from opening up properly its sockets and failing/quitting because of that.

I am not sure why this happens after initial bootup when the boot scripts start it up and not when I start it up myself or when watchdog restarts it. It could be something not ready yet and igmp proxy is trying to use it or maybe it is conflicting with something else in the bootup sequence of pfSense. I may compare later on why I do not have a similar behaviour on OPNsense, in case it clarifies why it happens here and open another ticket regarding this.

IGMP Proxy 0.3 seems to at least fix the kernel panic issue I was experiencing after bootup for now so I think we can probably close this ticket since the upgrade to 0.3 is now completed, thanks again for your help.

I think that I am currently having the same issues with igmp proxy since upgrading to pfsense 2.5.0:

2.5.0-RELEASE (amd64)
built on Tue Feb 16 08:56:29 EST 2021
FreeBSD 12.2-STABLE

When my system crashes I will have already received > 500 emails from the service watchdog informing me that "igmproxy stopped igmpproxy (IGMP proxy)"

I did experience the kernel panic/reboot/kernel panic continuous loop again during 2.5 DEV versions a few weeks ago, even with the latest igmp proxy version (after I confirmed that 0.3 was working here). It crashed and rebooted enough that the filesystem was corrupted too much to be able to boot anymore. I had to reinstall 2.5 cleanly and reimport my backup config file and reinstall packages. Since the reinstall, I did not had the kernel panic/reboot/panic reboot loop anymore.

So I wonder if there was something already corrupted from all the previous crashes (in 2.4 or 2.5 DEV testing) that may have led to this issue. I was running 2.4 latest at the time that igmp proxy first started kernel panics/boot. I always had been able to fsck the filesystem and get it back under control after some fiddling and also disabling igmp proxy so that I could at least get Internet back up but without IPTV. I upgraded to 2.5-dev on top of this 2.4 system. 2.5-dev showed the same issue (panic/reboot loop) with the previous version of igmp proxy. Version 0.3 seems to have fixed that for me except on 1 occasion where the panic/reboot corrupted the filesystem on 2.5-DEV and I decided to be sure that the filesystem was clean and I simply reinstalled everything (USB key installation, import config file from current installation in the installation menu, took me about 2 minutes to do and the system was clean and already configured like it was). No issue since then (fingers crossed, I hope I didn't jinxed myself here).

I have those packages installed as well (in case it may explain the current issue).

- acme
- apcupsd
- Avahi
- mailreport
- Service_Watchdog

Watchdog is installed to make sure igmp proxy get restarted if it crashes.

The only time IGMP proxy seems to crash now is on initial bootup and I still don't know why it happens.

"MC-Router API already in use; Errno(48): Address already in use"

But once watchdog restarts it, it works without any issue.

Can you provide us some logs (IGMP proxy seems to log to System/Routing and sometimes also under System) of IGMP proxy crashing ?

The only watchdog logs I have when is this on my side when igmp proxy crashes on startup (system log)

Feb 19 19:50:00 php 20457 servicewatchdog_cron.php: Started IGMP proxy service.
Feb 19 19:50:00 php 20457 servicewatchdog_cron.php: Service Watchdog detected service igmpproxy stopped. Restarting igmpproxy (IGMP proxy)

And this is what shows up when IGMP proxy gets started on initial bootup (with some context around the startup)

Feb 19 19:49:41 avahi-daemon 9677 avahi-daemon 0.8 starting up.
Feb 19 19:49:41 avahi-daemon 9677 Successfully dropped root privileges.
Feb 19 19:49:41 avahi-daemon 9677 Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Feb 19 19:49:41 php-fpm 84824 /rc.start_packages: Starting service avahi
Feb 19 19:49:41 php-fpm 84824 /rc.start_packages: Restarting/Starting all packages.
Feb 19 19:49:41 kernel done.
Feb 19 19:49:41 syslogd kernel boot file is /boot/kernel/kernel
Feb 19 19:49:41 syslogd exiting on signal 15
Feb 19 19:49:41 kernel done.
Feb 19 19:49:41 php 625 rc.bootup: Started IGMP proxy service.
Feb 19 19:49:41 php 625 rc.bootup: Creating rrd update script
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: Started IGMP proxy service.
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: Creating rrd update script
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: WARNING: Setting i_dont_care_about_security_and_use_aggressive_mode_psk option because a phase 1 is configured using aggressive mode with pre-shared keys. This is not a secure configuration.
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: Forcefully reloading IPsec

So yeah, I've been running about 10 days without issue (the reboot was to update to the latest 2.5 release).

Also, when you say you system crashes, you mean it kernel panics and reboots ? If so, can you upload the logs of that, like I did at the beginning of this ticket.