pfSense

There must be some other required component to replicate this. I've not seen a panic like this on the 6100 at my edge and I've been pushing every bit of my edge traffic through it consistently, a decent chunk of it going through IPv6 (with and without NPt).

I was able to download that entire Linux ISO torrent without error.

I have reinstalled the 6100 with the 23.01 to make sure that the issue is not linked to the upgrade.
I got the same result so i am back to the 22.05.

I will create a boot environment where i will upgrade to 23.01 and put it back to factory defaults.
Then I will try to rebuild my configuration step by step and do some tests.
Maybe I will be able to find the parameters that create the crash.

Bruno Dambrine wrote in #note-2:

I have reinstalled the 6100 with the 23.01 to make sure that the issue is not linked to the upgrade.
I got the same result so i am back to the 22.05.

I will create a boot environment where i will upgrade to 23.01 and put it back to factory defaults.
Then I will try to rebuild my configuration step by step and do some tests.
Maybe I will be able to find the parameters that create the crash.

Hi guys.

I think I posted the same issue on NON-Netgate hardware in main forum - https://forum.netgate.com/topic/178613/23-01-crashing-frequently-ipsec-connections-constantly-dropping-and-respawning-unable-to-access-http-over-vpn-address-constantly-times-out

Looks like the same panic error (I had uploaded the dump files in that message - maybe related??)

Hi.

I have rebuild the configuration and I may have some useful information.

First of all, some information on how I am connected to Internet. I have an optic fiber and at the end the Internet Service Provider box.
Behind the box there is a TV decoder and the Netgate 6100. All my network is behind the Netgate 6100 (Switches, PC, NAS, ...).

By rebuilding my configuration, I found that the crash comes with the NAT rules.
I downloads torrents (linux images, ...) with my QNAP NAS. So I use the QNAP Download Station for that (a software provided by QNAP and installed in the NAS).
The Download Station use port 6881 to 6889 for incoming TCP connection (to speed up download and seeding : the ports depends on the torrent client but it is a common practice for torrent client).
So I have done a NAT configuration on the ISP box to forward port 6881 to 6889 to the Netgate 6100.
And I have done a NAT configuration on the Netgate 6100 to forward port 6881 to 6889 to the NAS.
If I disabled the two rules (IPv4 and IPv6) in the 6100. I don't have the crash.
If I keep them but I disabled the rules in the ISP box. I still don't have the crash.
So it seems that the crash is linked to incoming connections not the configuration itself.
In my configuration, I also have a NAT on the port 443 for a HTTPS server in my network. But it does not provide any crash.
The main difference between the 443 NAT and the 6881-6889 NAT is the number of connections.
On the 443 NAT, I have only few connections but when I download some torrents, I have a lot connections on the 6881-6889 NAT.

My guess is the issue is about the NAT. Few connections is OK but a lot of connections leads to a crash.
So the downloads just create the conditions for the crash. The issue may appear on any situation where there are many connections on a NAT.

I hope these information can help solving the issue.

@Jim
Maybe during your tests, you could not have any incoming connection (firewall, ...). As I said earlier when I disable the NAT, everything is fine. It is just slower.

Thank you for the information.

I got an unexpected crash but I forgot that I have another NAT rule (the 443 NAT rule)...
So I switch back to 22.05 while you are working on a fix.

This issue isn't related to IPv4 NAT, so your NAT rules will not matter.

See #14092 as well, because this is almost certainly that issue. The fix is pending review upstream, and will likely land in snapshot builds later this week.

#14092 is not public, so it's impossible to check what that one is about and what will trigger it.

Sorry, I missed that.

I believe I understand the issue. Briefly put, pf_refragment6() ends up calling ip6_forward() for traffic in the output (so not forwarding) path, and ip6_forward() assumes that m->m_pkthdr.rcvif is set, which is not the case for output traffic.

This fixes the panic: https://reviews.freebsd.org/D39061 (and subsequent reviews fix link-local functionality and add a test case).

Sorry, I have two questions.

1 - Can I install the last snapshop of pfsense CE on my netgate 6100 as I do with pfsense+ ?

2 - Will i be able to reload the conf from the pfsense+ 22.05 or do I have to rebuild the conf ?

Bruno Dambrine wrote in #note-12:

1 - Can I install the last snapshop of pfsense CE on my netgate 6100 as I do with pfsense+ ?

It may technically be possible to some extent but not something I'd recommend, we do not test that or make sure it works overall. It would be missing support for various aspects of the system, though basic functionality may be there.

2 - Will i be able to reload the conf from the pfsense+ 22.05 or do I have to rebuild the conf ?

For compatibility between versions, the important piece is the "config revision" -- you can always import a config that is the same or older revision to a newer version, but you can't go backward. See https://docs.netgate.com/pfsense/en/latest/releases/versions.html for a table with which versions have which config revisions.

Eventually we'll have 23.05 snapshots public but at the moment we're still working on things quite heavily so they aren't generally available yet.

I'm not proficient with FreeBSD package management so this is probably a dumb question, but is there any way to drop a kernel with this fix onto an existing 23.01 system?

Jim Pingle wrote in #note-13:

Bruno Dambrine wrote in #note-12:

1 - Can I install the last snapshop of pfsense CE on my netgate 6100 as I do with pfsense+ ?

It may technically be possible to some extent but not something I'd recommend, we do not test that or make sure it works overall. It would be missing support for various aspects of the system, though basic functionality may be there.

2 - Will i be able to reload the conf from the pfsense+ 22.05 or do I have to rebuild the conf ?

For compatibility between versions, the important piece is the "config revision" -- you can always import a config that is the same or older revision to a newer version, but you can't go backward. See https://docs.netgate.com/pfsense/en/latest/releases/versions.html for a table with which versions have which config revisions.

Eventually we'll have 23.05 snapshots public but at the moment we're still working on things quite heavily so they aren't generally available yet.

Thanks for the explanation.

I will wait for a public snapshot of the 23.05.

This evening, I have installed the last beta of 23.05 on my 6100 and done some tests.
Currently no crash.

Thanks.

There are more details about this issue and specifics of how to easily reproduce it over on #14092 which is now public since we released 23.05 with the fix included.

This is the same root cause, though we didn't close it out as a duplicate since it was generating useful feedback.