pfSense

I saw this occur on a 7100 that had two bridged ixl interfaces for an add in card on 21.05.2, so it may affect basically everything from 2.4.5p1 to 22.01, potentially.

Same happened on 2.6.0 with Intel x710-T4 multiple times now.
Updating the nvme from 8.15 to latest 8.60 didn't fix the issue. Replacing the card with another X710 didn't help either.

sysctl -a | grep dev.ixl.0 | grep fw
dev.ixl.0.fw_lldp: 1
dev.ixl.0.fw_version: fw 8.6.68629 api 1.15 nvm 8.60 etid 8000bd5a oem 1.268.0

sysctl -a | grep dev.ixl.0.%desc
dev.ixl.0.%desc: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T - 2.3.1-k

Seems to only affect one port of the 4 ports, seems to be the one with the most traffic.

TSO is disabled by the checkbox and System => Advanced => Tunable => net.inet.tcp.tso set to 0

Christoph Vieten wrote in #note-2:

Same happened on 2.6.0 with Intel x710-T4 multiple times now.
Updating the nvme from 8.15 to latest 8.60 didn't fix the issue. Replacing the card with another X710 didn't help either.

sysctl -a | grep dev.ixl.0 | grep fw
dev.ixl.0.fw_lldp: 1
dev.ixl.0.fw_version: fw 8.6.68629 api 1.15 nvm 8.60 etid 8000bd5a oem 1.268.0

sysctl -a | grep dev.ixl.0.%desc
dev.ixl.0.%desc: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T - 2.3.1-k

Seems to only affect one port of the 4 ports, seems to be the one with the most traffic.

TSO is disabled by the checkbox and System => Advanced => Tunable => net.inet.tcp.tso set to 0

Christoph,

Were you running a bridge in your configuration like the original bug report seems to suggest is the root cause?

Kris Phillips wrote in #note-3:

Christoph Vieten wrote in #note-2:

Same happened on 2.6.0 with Intel x710-T4 multiple times now.
Updating the nvme from 8.15 to latest 8.60 didn't fix the issue. Replacing the card with another X710 didn't help either.

sysctl -a | grep dev.ixl.0 | grep fw
dev.ixl.0.fw_lldp: 1
dev.ixl.0.fw_version: fw 8.6.68629 api 1.15 nvm 8.60 etid 8000bd5a oem 1.268.0

sysctl -a | grep dev.ixl.0.%desc
dev.ixl.0.%desc: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T - 2.3.1-k

Seems to only affect one port of the 4 ports, seems to be the one with the most traffic.

TSO is disabled by the checkbox and System => Advanced => Tunable => net.inet.tcp.tso set to 0

Christoph,

Were you running a bridge in your configuration like the original bug report seems to suggest is the root cause?

Hi Kris,

no, were aren't running a bridge at all. But we are running approx. 20 vlan interfaces on the port that is affected.
Looks like when the issue occurs, you cannot switch to other physical ports (we have three of those X710 quad port cards in use) of any other adapter as well.
But the other ports in use (e.g. some 10g ports are configured without vlan assignments or have a smaller number of vlans) aren't affected by that driver / firmware stuck issue so can still be used.

Last time when the issue occurred, we migrated the top traffic vlan interfaces to separate ports resulting in a longer uptime until yesterday.

Did someone try the latest FreeBSD driver yet?
https://pkg.freebsd.org/FreeBSD:12:amd64/latest/All/intel-ix-kmod-3.3.24.pkg

Christoph Vieten wrote in #note-5:

Kris Phillips wrote in #note-3:

Christoph Vieten wrote in #note-2:

Same happened on 2.6.0 with Intel x710-T4 multiple times now.
Updating the nvme from 8.15 to latest 8.60 didn't fix the issue. Replacing the card with another X710 didn't help either.

sysctl -a | grep dev.ixl.0 | grep fw
dev.ixl.0.fw_lldp: 1
dev.ixl.0.fw_version: fw 8.6.68629 api 1.15 nvm 8.60 etid 8000bd5a oem 1.268.0

sysctl -a | grep dev.ixl.0.%desc
dev.ixl.0.%desc: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T - 2.3.1-k

Seems to only affect one port of the 4 ports, seems to be the one with the most traffic.

TSO is disabled by the checkbox and System => Advanced => Tunable => net.inet.tcp.tso set to 0

Christoph,

Were you running a bridge in your configuration like the original bug report seems to suggest is the root cause?

Hi Kris,

no, were aren't running a bridge at all. But we are running approx. 20 vlan interfaces on the port that is affected.
Looks like when the issue occurs, you cannot switch to other physical ports (we have three of those X710 quad port cards in use) of any other adapter as well.
But the other ports in use (e.g. some 10g ports are configured without vlan assignments or have a smaller number of vlans) aren't affected by that driver / firmware stuck issue so can still be used.

Last time when the issue occurred, we migrated the top traffic vlan interfaces to separate ports resulting in a longer uptime until yesterday.

Did someone try the latest FreeBSD driver yet?
https://pkg.freebsd.org/FreeBSD:12:amd64/latest/All/intel-ix-kmod-3.3.24.pkg

Hello Christoph,

I don't see any notes that it's been tested for this particular issue. However, the Intel ix driver was updated in 22.05. Have you tested to see if this issue is gone in the latest RC? We expect 22.05 to be released very soon, so might be worth a re-test on the latest.

Good afternoon.

We have the same problem with our 1541.

We are in version 23.01.

The problem is that sometimes the IXL3 interface begins to block all traffic until we restart the firewall or the IXL3 interface.
In the system record we see the next line just when this happens:
IXL3: Malicious Driver Detection event 2 on TX queue 1152, pf number 3 (PF-3)

We have already tried to disable TSO (both in "System/Advanced/Networking" and "Tunables System"), but this has not solved the problem.

IXL0 and IXL3 interfaces are on a bridge.

Greetings.

Good afternoon.

We have tried updating the driver, disabling TSO, increasing queues, changing the interface... Without any results, we continue with daily crashes.

Mar 6 15:25:35 pfs01 kernel: ixl3: Malicious Driver Detection event 2 on TX queue 1152, pf number 3 (PF-3)
Mar 7 13:50:24 pfs01 kernel: ixl3: Malicious Driver Detection event 2 on TX queue 1152, pf number 3 (PF-3)
Mar 7 14:48:39 pfs01 kernel: ixl3: Malicious Driver Detection event 2 on TX queue 1152, pf number 3 (PF-3)
Mar 8 08:29:40 pfs01 kernel: ixl3: Malicious Driver Detection event 2 on TX queue 1152, pf number 3 (PF-3)
Mar 8 11:38:56 pfs01 kernel: ixl3: Malicious Driver Detection event 2 on TX queue 1152, pf number 3 (PF-3)
Mar 9 13:48:16 pfs01 kernel: ixl2: Malicious Driver Detection event 2 on TX queue 768, pf number 2 (PF-2)
Mar 10 12:20:29 pfs01 kernel: ixl2: Malicious Driver Detection event 2 on TX queue 768, pf number 2 (PF-2)
Mar 10 16:30:13 pfs01 kernel: ixl2: Malicious Driver Detection event 2 on TX queue 768, pf number 2 (PF-2)
Mar 10 16:39:17 pfs01 kernel: ixl2: Malicious Driver Detection event 2 on TX queue 768, pf number 2 (PF-2)

The data of our NIC are the following:
dev.ixl.0.fw_version: fw 6.0.48442 api 1.7 nvm 6.01 etid 8000351b oem 0.0.0
dev.ixl.0.iflib.driver_version: 2.3.2-k
dev.ixl.0.%pnpinfo: vendor=0x8086 device=0x1572 subvendor=0x8086 subdevice=0x0001 class=0x020000
dev.ixl.0.%location: slot=0 function=0 dbsf=pci0:5:0:0 handle=\_SB_.PCI0.BR3A.H000
dev.ixl.0.%driver: ixl
dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.2-k

The 1541 is at version 23.01-RELEASE.

I don't understand that the answer from NETGATE support is the following, I find it unfortunate:

Marcos
4 days ago

Hello,

It seems like you are experiencing the issue reported here:
https://redmine.pfsense.org/issues/13003

Unfortunately we do not have a workaround to provide. You may check to see if there are newer NIC firmwares available, or potentially use another vendor / NIC that uses different drivers. In either case, I suggest adding your report as a comment there and clarify that this is still an issue on 23.01 (I assume that's correct given that's what was specified on the ticket).

- - -
Marcos M.
Netgate Global Support

Today we have had a crash with the "Malicious Driver Detection" event at 10:00:26 Colombia time:
Mar 13 10:00:26 kernel ixl2: Malicious Driver Detection event 2 on TX queue 768, pf number 2 (PF-2)

The system LOG does not reflect anything conclusive:
Mar 13 10:05:59 check_reload_status 390 Reloading filter
Mar 13 10:05:59 php-fpm 360 /rc.linkup: Ignoring link event for bridge member without IP address configuration
Mar 13 10:05:58 check_reload_status 390 Reloading filter
Mar 13 10:05:58 php-fpm 360 /rc.linkup: Ignoring link event for bridge member without IP address configuration
Mar 13 10:05:58 kernel ixl2: link state changed to UP
Mar 13 10:05:58 kernel ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Mar 13 10:05:58 check_reload_status 390 Linkup starting ixl2
Mar 13 10:05:57 kernel ixl2: link state changed to DOWN
Mar 13 10:05:57 check_reload_status 390 Linkup starting ixl2
Mar 13 10:05:38 php-fpm 6960 /index.php: Successful login for user 'admin' from: 47.62.22.86 (Local Database)
Mar 13 10:05:35 php-fpm 6960 /index.php: Session timed out for user 'admin' from: 47.62.22.86 (Local Database)
Mar 13 10:00:26 kernel ixl2: Malicious Driver Detection event 2 on TX queue 768, pf number 2 (PF-2)
Mar 13 08:29:00 sshguard 47986 Now monitoring attacks.
Mar 13 08:29:00 sshguard 81272 Exiting on signal.

Greetings.