Project

General

Profile

Bug #11330

IGMP Proxy upgrade to latest version

Added by Patrick Monfette 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Category:
IGMP Proxy
Target version:
Start date:
01/28/2021
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.5.0
Affected Architecture:
amd64
Release Notes:
Default

Description

IGMP Proxy has been updated to 0.3

https://github.com/pali/igmpproxy/releases/tag/0.3

Would it be possible to include this release in 2.5 ?

I'm experiencing issues with IPTV and 2.5 that are leading to crash / reboot loop on boot when igmp proxy starts or gets enabled. Leaving IGMP proxy disabled allows pfSense to properly start and remain stable. Once started, if I enable IGMP proxy, there is a kernel panic and reboot and it keeps happening until I disable IGMP proxy or disconnect the IPTV network to prevent any traffic on it.

I'm thinking this may be related to IGMP proxy or maybe to some underlying multicast issue (either caused by it IGMP proxy or the kernel itself).

I've tried OPNsense to compare and with the exact same setup: IPTV, igmp proxy (they provide 0.3 in their packages), same firewall rules, etc... it is not happening and everything seems stable.

So far, IGMP proxy seems the culprit but I can't explain why it would lead to a kernel panic, unless this is some side effect on multicast once traffic starts flowing after igmp proxy did his job.

info.0 (410 Bytes) info.0 Patrick Monfette, 01/29/2021 01:29 AM
textdump.tar.0 (154 KB) textdump.tar.0 Patrick Monfette, 01/29/2021 01:29 AM

History

#1 Updated by Patrick Monfette 2 months ago

Attached are the crashdump files.

You'll also see in the backlogs that because it rebooted so often in loop that I also had to do a fsck a few times to clean things up. Once the filesystem was clean, I did a last boot, it crashed again and here is the crashdump file of this time.

I disabled igmp proxy, rebooted and all is good now, system is stable (but I don't have IPTV working since igmp proxy is disabled).

Like I said, it could be igmp proxy or it could be when multicast starts flowing once igmp proxy does its job. Not sure which ones triggers the kernel panic.

It seems to happen more often right after bootup, usually almost every time after bootup. If, for some reason, it doesn't crash and starts to work, I'm usually good, it will continue to work without crashing. But if I reboot, most of the time, right after the interfaces are up and multicast starts flowing, kernel panic happens.

#2 Updated by Patrick Monfette 2 months ago

I decided to test the OPNsense igmp proxy 0.3 package by force installing it over the version (0.2.1) that comes with pfSense 2.5

like this: pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/21.1/latest/Latest/igmpproxy.txz

I did about 12 reboots with this version and no kernel panic so far. So, I guess igmp proxy is really the culprit behind those kernel panics (or the one that triggers something that leads to a kernel panic) and version 0.3 fixes that behaviour.

I do however still have an issue. pfSense does boot properly, starts everything but when it first starts IGMP proxy, it gives this error (on every bootup):

Jan 29 03:50:41 [myhostname] igmpproxy40587: MC-Router API already in use; Errno(48): Address already in use

It seems like igmp proxy dies right after that error.

Not sure why this happens or what may conflict with it. Since I have installed service watchdog and configured it to monitor and restart igmpproxy in case of failure, igmpproxy gets restarted and then continues to work without any issue after that.

So, 0.3 seems to help or fix the kernel panic issue at least which is good so I think it would be a good idea to release it with 2.5 to prevent kernel panics at least. igmpproxy still quits on first startup however after initial boot but if you restart it manually (or by some other mean like service watchdog), it will restart without any issue or error and continue to work after that.

#3 Updated by Renato Botelho 2 months ago

  • Status changed from New to Feedback
  • Assignee set to Renato Botelho
  • Private changed from Yes to No

Version 0.3 was cherry-picked from FreeBSD and will be available on next round of snapshots

#4 Updated by Patrick Monfette 2 months ago

Confirmed, I see 0.3 now, thank you.

I still need to run watchdog to get the process restarted after initial bootup because of this error:

Feb  3 22:55:33 xxxxxx igmpproxy [70274]: MC-Router API already in use; Errno(48): Address already in use

It seems to happen when igmp proxy tries to start up here

    switch ( Err = enableMRouter() ) {
    case 0: break;
    case EADDRINUSE: my_log( LOG_ERR, EADDRINUSE, "MC-Router API already in use" ); break;
    default: my_log( LOG_ERR, Err, "MRT_INIT failed" );
    }

while doing this

int enableMRouter(void)
{
    int Va = 1;

    if ( (MRouterFD  = socket(AF_INET, SOCK_RAW, IPPROTO_IGMP)) < 0 )
        my_log( LOG_ERR, errno, "IGMP socket open" );

    if ( setsockopt( MRouterFD, IPPROTO_IP, MRT_INIT,
                     (void *)&Va, sizeof( Va ) ) )
        return errno;

    return 0;
}

For some reason, this seems to fail and triggers this error and it then quits, only on initial pfSense bootup sequence. It seems like there is something preventing igmp proxy from opening up properly its sockets and failing/quitting because of that.

I am not sure why this happens after initial bootup when the boot scripts start it up and not when I start it up myself or when watchdog restarts it. It could be something not ready yet and igmp proxy is trying to use it or maybe it is conflicting with something else in the bootup sequence of pfSense. I may compare later on why I do not have a similar behaviour on OPNsense, in case it clarifies why it happens here and open another ticket regarding this.

IGMP Proxy 0.3 seems to at least fix the kernel panic issue I was experiencing after bootup for now so I think we can probably close this ticket since the upgrade to 0.3 is now completed, thanks again for your help.

#5 Updated by Renato Botelho 2 months ago

  • Status changed from Feedback to Resolved

#6 Updated by simon lock about 1 month ago

I think that I am currently having the same issues with igmp proxy since upgrading to pfsense 2.5.0:

2.5.0-RELEASE (amd64)
built on Tue Feb 16 08:56:29 EST 2021
FreeBSD 12.2-STABLE

When my system crashes I will have already received > 500 emails from the service watchdog informing me that "igmproxy stopped igmpproxy (IGMP proxy)"

#7 Updated by Patrick Monfette about 1 month ago

I did experience the kernel panic/reboot/kernel panic continuous loop again during 2.5 DEV versions a few weeks ago, even with the latest igmp proxy version (after I confirmed that 0.3 was working here). It crashed and rebooted enough that the filesystem was corrupted too much to be able to boot anymore. I had to reinstall 2.5 cleanly and reimport my backup config file and reinstall packages. Since the reinstall, I did not had the kernel panic/reboot/panic reboot loop anymore.

So I wonder if there was something already corrupted from all the previous crashes (in 2.4 or 2.5 DEV testing) that may have led to this issue. I was running 2.4 latest at the time that igmp proxy first started kernel panics/boot. I always had been able to fsck the filesystem and get it back under control after some fiddling and also disabling igmp proxy so that I could at least get Internet back up but without IPTV. I upgraded to 2.5-dev on top of this 2.4 system. 2.5-dev showed the same issue (panic/reboot loop) with the previous version of igmp proxy. Version 0.3 seems to have fixed that for me except on 1 occasion where the panic/reboot corrupted the filesystem on 2.5-DEV and I decided to be sure that the filesystem was clean and I simply reinstalled everything (USB key installation, import config file from current installation in the installation menu, took me about 2 minutes to do and the system was clean and already configured like it was). No issue since then (fingers crossed, I hope I didn't jinxed myself here).

I have those packages installed as well (in case it may explain the current issue).

- acme
- apcupsd
- Avahi
- mailreport
- Service_Watchdog

Watchdog is installed to make sure igmp proxy get restarted if it crashes.

The only time IGMP proxy seems to crash now is on initial bootup and I still don't know why it happens.

"MC-Router API already in use; Errno(48): Address already in use"

But once watchdog restarts it, it works without any issue.

Can you provide us some logs (IGMP proxy seems to log to System/Routing and sometimes also under System) of IGMP proxy crashing ?

The only watchdog logs I have when is this on my side when igmp proxy crashes on startup (system log)

Feb 19 19:50:00 php 20457 servicewatchdog_cron.php: Started IGMP proxy service.
Feb 19 19:50:00 php 20457 servicewatchdog_cron.php: Service Watchdog detected service igmpproxy stopped. Restarting igmpproxy (IGMP proxy)

And this is what shows up when IGMP proxy gets started on initial bootup (with some context around the startup)

Feb 19 19:49:41 avahi-daemon 9677 avahi-daemon 0.8 starting up.
Feb 19 19:49:41 avahi-daemon 9677 Successfully dropped root privileges.
Feb 19 19:49:41 avahi-daemon 9677 Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Feb 19 19:49:41 php-fpm 84824 /rc.start_packages: Starting service avahi
Feb 19 19:49:41 php-fpm 84824 /rc.start_packages: Restarting/Starting all packages.
Feb 19 19:49:41 kernel done.
Feb 19 19:49:41 syslogd kernel boot file is /boot/kernel/kernel
Feb 19 19:49:41 syslogd exiting on signal 15
Feb 19 19:49:41 kernel done.
Feb 19 19:49:41 php 625 rc.bootup: Started IGMP proxy service.
Feb 19 19:49:41 php 625 rc.bootup: Creating rrd update script
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: Started IGMP proxy service.
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: Creating rrd update script
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: WARNING: Setting i_dont_care_about_security_and_use_aggressive_mode_psk option because a phase 1 is configured using aggressive mode with pre-shared keys. This is not a secure configuration.
Feb 19 19:49:40 php-fpm 565 /rc.newwanip: Forcefully reloading IPsec

So yeah, I've been running about 10 days without issue (the reboot was to update to the latest 2.5 release).

Also, when you say you system crashes, you mean it kernel panics and reboots ? If so, can you upload the logs of that, like I did at the beginning of this ticket.

Also available in: Atom PDF