pfSense » pfSense Plus

My post on the netgate forum, still no unanswer:
https://forum.netgate.com/topic/175096/ipsec-with-qat-low-performance-netgate-cpic-8955?_=1677835676706

Meanwhile I upgraded the operating system to:
pfsense+ 23.01-RELEASE (amd64) / FreeBSD 14.0-CURRENT / CPIC-8955 hardware accelerator

The exact same thing happens here:
https://forum.netgate.com/topic/162176/openssl-qat

AES-NI CPU Crypto: Yes (active)
QAT Crypto: Yes (active)

/root: openssl engine
(devcrypto) /dev/crypto engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support

/root: openssl speed -engine rdrand -evp aes-128-gcm
engine "rdrand" set.
Doing aes-128-gcm for 3s on 16 size blocks: 109473266 aes-128-gcm's in 3.15s
Doing aes-128-gcm for 3s on 64 size blocks: 59620644 aes-128-gcm's in 3.06s
Doing aes-128-gcm for 3s on 256 size blocks: 37145965 aes-128-gcm's in 3.05s
Doing aes-128-gcm for 3s on 1024 size blocks: 12758891 aes-128-gcm's in 3.07s
Doing aes-128-gcm for 3s on 8192 size blocks: 1961291 aes-128-gcm's in 3.06s
Doing aes-128-gcm for 3s on 16384 size blocks: 1004601 aes-128-gcm's in 3.09s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-gcm     556330.64k  1245949.78k  3121023.03k  4255301.17k  5246333.35k  5320204.54k

/root: openssl speed -engine devcrypto -evp aes-128-gcm
engine "devcrypto" set.
Doing aes-128-gcm for 3s on 16 size blocks: 109588628 aes-128-gcm's in 3.09s
Doing aes-128-gcm for 3s on 64 size blocks: 58764133 aes-128-gcm's in 3.08s
Doing aes-128-gcm for 3s on 256 size blocks: 36989212 aes-128-gcm's in 3.08s
Doing aes-128-gcm for 3s on 1024 size blocks: 12517930 aes-128-gcm's in 3.03s
Doing aes-128-gcm for 3s on 8192 size blocks: 1892616 aes-128-gcm's in 3.01s
Doing aes-128-gcm for 3s on 16384 size blocks: 962895 aes-128-gcm's in 3.02s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-gcm     566761.39k  1221816.69k  3076300.76k  4228737.43k  5154679.78k  5217925.52k

dmesg | grep qat
qat0: <Intel dh895xcc QuickAssist> mem 0xe0600000-0xe067ffff,0xefd40000-0xefd7ffff,0xefd00000-0xefd3ffff irq 32 at device 0.0 on pci2
qat_dh895xcc_fw: could not load firmware image, error 6
qat0: Failed to load UOF FW qat_dh895xcc_fw
qat0: Failed to load acceleration FW
qat0: Resetting device qat_dev0
qat0: Secondary bus reset
device_attach: qat0 attach returned 14
qat0: <Intel dh895xcc QuickAssist> mem 0xe0600000-0xe067ffff,0xefd40000-0xefd7ffff,0xefd00000-0xefd3ffff irq 32 at device 0.0 on pci2
qat0: qat_dev0 started 12 acceleration engines
qat0: FW version: 4.18.0
qat_ocf0: <QAT engine>

kldstat | grep qat
 2    1 0xffffffff83ba5000     81d8 qat.ko
 3    7 0xffffffff83bae000    7f320 qat_api.ko
 4    8 0xffffffff83c2e000    44280 qat_common.ko
 5    6 0xffffffff83c73000    1fe98 qat_hw.ko
11    1 0xffffffff84aef000   161f38 qat_dh895xcc_fw.ko
kldstat | grep aes
16    1 0xffffffff84c61000     a288 aesni.ko

Clients using AES_GCM_16 (128) PRF_HMAC_SHA1 MODP_2048, the maximum IPSEC speed reached is around 800Mb/s ( TOTAL , even with multiple clients, parallel connections)

Also, I want to specify that I tested this QAT adapter on another server with the same results, I also tested on a different server with much higher performance.

Attached you can find config file and some tests. More you can find in my post:
https://forum.netgate.com/topic/175096/ipsec-with-qat-low-performance-netgate-cpic-8955?_=1677835676706

Thanks!