Feature #14173
closed
QAT driver does not attach to QAT virtual function devices passed through to VM on Xeon D-2146NT
0%
Description
pfSense is virtualized under Linux.
Hypervisor:
- qemu-kvm
- i440fx (q35 doesn't work either)
- kernel 5.15.94 with qat17 driver version 4.20.0.00001
- QAT Virtual functions of a Xeon D-2146NT passed through.
pfSense Plus 23.01 VM:
$ dmesg | grep -i qat
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
$ cat /boot/loader.conf.local
qat_load="YES"
qat_c2xxx_fw_load="YES"
qat_c3xxx_fw_load="YES"
qat_c62x_fw_load="YES"
qat_d15xx_fw_load="YES"
$ pciconf -lv
. . .
none9@pci0:1:5:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processor
none10@pci0:1:6:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processor
none11@pci0:1:7:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processor
none12@pci0:1:8:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processor
none13@pci0:1:9:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processor
none14@pci0:1:10:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processor
none15@pci0:1:11:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processor
none16@pci0:1:12:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processor
This was also tested, with the same problem, when using FreeBSD-14.0-CURRENT-amd64-20230323-b5d43972e394-261711.iso as live CD.
Updated by Jim Pingle about 1 year ago
- Tracker changed from Bug to Feature
- Subject changed from QAT not working for Xeon D-2146NT QAT virtual functions passed through to the pfSense VM to QAT driver does not attach to QAT virtual function devices passed through to VM on Xeon D-2146NT
- Status changed from New to Needs Patch
- Affected Plus Version deleted (
23.01) - Affected Architecture deleted (
amd64)
If it fails on FreeBSD 14-CURRENT then it needs fixed upstream first and we can pull in the fix from there. It could be a matter of adding the VF device IDs to the driver but it may not be that simple.
Updated by name name about 1 year ago
Hi Jim,
thank you for looking into it.
I'm already in contact with the Intel QAT driver team, to see if the fault is on my end or what could be done to get it working.
As soon as I have further information, I'll make sure to share it.
Should someone else have the same hardware and know how to fix it, please let me know the working configuration.
Note: On the Linux host the qat_c62xvf driver is successfully loaded for the virtual functions and until libvirt unbinds them for passthrough, everything seems fine.
Updated by name name 12 months ago
I was informed by Intel that there will be no QAT support in FreeBSD 14.0 for any chips other than those belonging to QAT hardware generation 2, which is solely limited to the Intel Xeon Scalable 4th Gen platform.
According to Intel there are no plans to support FreeBSD 14 for QAT harware generation 1.x, which should mean systems based on the following processors/chipsets aren't supported:
Intel Atom C5000
Intel Atom P5000
Intel Xeon D1700
Intel Xeon D2300
Intel Xeon D2700
As these are all the newest platforms, some of which are just now becoming available for purchase, I guess it would have been better if you hadn't switched to FreeBSD 14.0 just yet.
Please ask Intel yourself and add the information to the documentation of pfSense.
Could you please let me know how the efforts for porting pfSense over to Linux are coming along?
https://fd.io/docs/vpp/v2101/events/summits/dpdksummit/2017/2017_11_15_dpdkvppandpfsense.html
Updated by Jim Pingle 12 months ago
We make sure that QAT works on hardware we sell: C3000, C2000, and the add-on CPIC cards in the Netgate shop. Support may not be from Intel but it's in FreeBSD 14. The documentation already mentions this list specifically: https://docs.netgate.com/pfsense/en/latest/hardware/cryptographic-accelerators.html#supported-devices - Beyond hardware we sell, we can't make any promises, and the docs already don't suggest that it would work anywhere but on what's listed.
That said, when I said "upstream" I meant FreeBSD directly and not Intel. Depending on the hardware involved, it may be a different response.
Also, on 23.05 you might find that IPsec-MB/IIMB (#14291) gives a sufficient performance boost that you won't necessarily need to rely on QAT, you'll just need to ensure you pass through the correct CPU type to the VM.
Updated by name name 12 months ago
Thank you for responding.
I'm aware of the hardware you are selling. Now that the next generation of Atom and Xeon-D CPUs are out, I guess it is only a question of time until the C2000, C3000 and Xeon-D 1500 series mainboards are no longer available. Supermicro A2SDi mainboards with Atom C3000 series CPUs have the EOL date Q1 2024. My guess would be that others will follow.
Considering that Intel told me they don't support QAT VFs on FreeBSD when using the in-kernel QAT driver, which would be FreeBSD upstream, and the fact that no out-of-tree driver for the above listed processors are planned for FreeBSD 14, a note in the documentation as warning for those who use your product, but perhaps not your hardware, would perhaps be something that you could agree to?
Also, the above CPUs will probably be interesting for lots of people looking into Wireguard VPN, as only QAT gen 3 and above provide hardware acceleration for ChaCha20-Poly1305.
For others reading this, although the systems with Atom C5000 and Xeon D 1700 processors are just now becoming available, Intel decided to only include QAT gen 2, which means that they can't accelerate Wireguard. QAT gen 2 is the version that is also in Atom C3000 processors.
The following all have QAT gen 3 and can therefore accelerate Wireguard: Intel Atom P5300, P5700, Intel Ice Lake-D 2700.