pfSense

Since upgrading my install to 2.4.3 I have noticed a decrease in the results I get from speedtest.net and fast.com
I spent a lot of time post upgrade fairly sure it must be something I have done, either a misconfiguration of my pfSense device, problems with my Wifi (even though I have also tested using fixed Ethernet) or my ISP having congestion etc issues.

To eventually rule it out though, I re-installed 2.4.2 and applied my backed up configuration and tested again, which restored the good performance that I previously had.

My setup is as follows:

• Physical Hardware: Mini PC with a Intel(R) Core(TM) i5-5250U CPU (latest 2018 microcode in use), 4 x I211 GigE Ports
• Running Proxmox Hypervisor (KVM)
• pfSense is running with 1Gb memory allocated
• pfSense is using VirtIO for Disc and Network
• PTI is disabled - both at the host level (using nopti on the Linux boot) and at the 2.4.3 guest level.
• ISP requires PPPoE
• I have a 100/20 Fibre Ethernet connection.
• The two extra Packages installed are Avahi and OpenVPN Export

In my Hypervisor I have two pfSense instances. I can shutdown 2.4.3 to boot 2.4.2 etc to do various tests, showing me the difference in performance between the two.

My configuration has FAIRQ enabled with the following settings:

 altq on pppoe0 fairq qlimit 1024  bandwidth 20000Kb queue {  qBulk,  qLow,  qMedium,  qHigh,  qVeryHigh,  qPriority  }
queue qBulk on pppoe0 priority 0 qlimit 455 fairq (  red  codel  )
queue qLow on pppoe0 priority 1 qlimit 256 fairq (  codel  , default  )
queue qMedium on pppoe0 priority 2 qlimit 256 fairq (  codel  )
queue qHigh on pppoe0 priority 3 qlimit 32 fairq (  codel  )
queue qVeryHigh on pppoe0 priority 4 qlimit 16
queue qPriority on pppoe0 priority 7 qlimit 8

 altq on vtnet1 fairq qlimit 8192  bandwidth 100000Kb queue {  qBulk,  qLow,  qMedium,  qHigh,  qVeryHigh,  qPriority  }
queue qBulk on vtnet1 priority 0 qlimit 456 fairq (  red  codel  )
queue qLow on vtnet1 priority 1 qlimit 256 fairq (  codel  , default  )
queue qMedium on vtnet1 priority 2 qlimit 256 fairq (  codel  )
queue qHigh on vtnet1 priority 3 qlimit 32 fairq (  codel  )
queue qVeryHigh on vtnet1 priority 4 qlimit 16
queue qPriority on vtnet1 priority 7 qlimit 8

When running speedtest.net with 2.4.3, a single stream will achieve approximately 50-60Mb/s of throughput. This is tested with no other load on the network.
When running speedtest.net with 2.4.2, a single stream will achieve approximately 80-90Mb/s of throughput.

I initially thought maybe there were problems with FAIRQ in 2.4.3 so I changed from FAIRQ to PRIQ, but I still see similar results.
It seems that with ALTQ enabled, performance is worse with 2.4.3.

If I disable ALTQ fully, I achieve similar performance on both 2.4.2/3, approx 94-95Mb/s which I figure is probably 100Mb/s minus PPPoE overheads.

Here is the output of my boot on 2.4.3

Apr 14 14:01:41 trogdor syslogd: kernel boot file is /boot/kernel/kernel
Apr 14 14:01:41 trogdor kernel: Copyright (c) 1992-2017 The FreeBSD Project.
Apr 14 14:01:41 trogdor kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Apr 14 14:01:41 trogdor kernel: The Regents of the University of California. All rights reserved.
Apr 14 14:01:41 trogdor kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Apr 14 14:01:41 trogdor kernel: FreeBSD 11.1-RELEASE-p7 #10 r313908+986837ba7e9(RELENG_2_4): Mon Mar 26 18:08:25 CDT 2018
Apr 14 14:01:41 trogdor kernel: root@buildbot2.netgate.com:/builder/ce-243/tmp/obj/builder/ce-243/tmp/FreeBSD-src/sys/pfSense amd64
Apr 14 14:01:41 trogdor kernel: FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1)
Apr 14 14:01:41 trogdor kernel: VT(vga): text 80x25
Apr 14 14:01:41 trogdor kernel: CPU: Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz (1596.42-MHz K8-class CPU)
Apr 14 14:01:41 trogdor kernel: Origin="GenuineIntel"  Id=0x306d4  Family=0x6  Model=0x3d  Stepping=4
Apr 14 14:01:41 trogdor kernel: Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
Apr 14 14:01:41 trogdor kernel: Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
Apr 14 14:01:41 trogdor kernel: AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Apr 14 14:01:41 trogdor kernel: AMD Features2=0x121<LAHF,ABM,Prefetch>
Apr 14 14:01:41 trogdor kernel: Structured Extended Features=0x1c07ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP>
Apr 14 14:01:41 trogdor kernel: Structured Extended Features3=0x4000000<IBPB>
Apr 14 14:01:41 trogdor kernel: XSAVE Features=0x1<XSAVEOPT>
Apr 14 14:01:41 trogdor kernel: Hypervisor: Origin = "KVMKVMKVM" 
Apr 14 14:01:41 trogdor kernel: real memory  = 1073741824 (1024 MB)
Apr 14 14:01:41 trogdor kernel: avail memory = 983093248 (937 MB)
Apr 14 14:01:41 trogdor kernel: Event timer "LAPIC" quality 600
Apr 14 14:01:41 trogdor kernel: ACPI APIC Table: <BOCHS  BXPCAPIC>
Apr 14 14:01:41 trogdor kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Apr 14 14:01:41 trogdor kernel: FreeBSD/SMP: 1 package(s) x 2 core(s)
Apr 14 14:01:41 trogdor kernel: random: unblocking device.
Apr 14 14:01:41 trogdor kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
Apr 14 14:01:41 trogdor kernel: SMP: AP CPU #1 Launched!
Apr 14 14:01:41 trogdor kernel: random: entropy device external interface
Apr 14 14:01:41 trogdor kernel: wlan: mac acl policy registered
Apr 14 14:01:41 trogdor kernel: kbd1 at kbdmux0
Apr 14 14:01:41 trogdor kernel: netmap: loaded module
Apr 14 14:01:41 trogdor kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff81162bc0, 0) error 19
Apr 14 14:01:41 trogdor kernel: random: registering fast source Intel Secure Key RNG
Apr 14 14:01:41 trogdor kernel: random: fast provider: "Intel Secure Key RNG" 
Apr 14 14:01:41 trogdor kernel: nexus0
Apr 14 14:01:41 trogdor kernel: vtvga0: <VT VGA driver> on motherboard
Apr 14 14:01:41 trogdor kernel: cryptosoft0: <software crypto> on motherboard
Apr 14 14:01:41 trogdor kernel: padlock0: No ACE support.
Apr 14 14:01:41 trogdor kernel: acpi0: <BOCHS BXPCRSDT> on motherboard
Apr 14 14:01:41 trogdor kernel: acpi0: Power Button (fixed)
Apr 14 14:01:41 trogdor kernel: cpu0: <ACPI CPU> on acpi0
Apr 14 14:01:41 trogdor kernel: cpu1: <ACPI CPU> on acpi0
Apr 14 14:01:41 trogdor kernel: atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Apr 14 14:01:41 trogdor kernel: Event timer "RTC" frequency 32768 Hz quality 0
Apr 14 14:01:41 trogdor kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Apr 14 14:01:41 trogdor kernel: Timecounter "HPET" frequency 100000000 Hz quality 950
Apr 14 14:01:41 trogdor kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Apr 14 14:01:41 trogdor kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
Apr 14 14:01:41 trogdor kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Apr 14 14:01:41 trogdor kernel: pci0: <ACPI PCI bus> on pcib0
Apr 14 14:01:41 trogdor kernel: isab0: <PCI-ISA bridge> at device 1.0 on pci0
Apr 14 14:01:41 trogdor kernel: isa0: <ISA bus> on isab0
Apr 14 14:01:41 trogdor kernel: atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe0c0-0xe0cf at device 1.1 on pci0
Apr 14 14:01:41 trogdor kernel: ata0: <ATA channel> at channel 0 on atapci0
Apr 14 14:01:41 trogdor kernel: ata1: <ATA channel> at channel 1 on atapci0
Apr 14 14:01:41 trogdor kernel: uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xe040-0xe05f irq 11 at device 1.2 on pci0
Apr 14 14:01:41 trogdor kernel: usbus0 on uhci0
Apr 14 14:01:41 trogdor kernel: usbus0: 12Mbps Full Speed USB v1.0
Apr 14 14:01:41 trogdor kernel: pci0: <bridge> at device 1.3 (no driver attached)
Apr 14 14:01:41 trogdor kernel: vgapci0: <VGA-compatible display> mem 0xfd000000-0xfdffffff,0xfea90000-0xfea90fff at device 2.0 on pci0
Apr 14 14:01:41 trogdor kernel: vgapci0: Boot video device
Apr 14 14:01:41 trogdor kernel: virtio_pci0: <VirtIO PCI Balloon adapter> port 0xe060-0xe07f mem 0xfe400000-0xfe403fff irq 11 at device 3.0 on pci0
Apr 14 14:01:41 trogdor kernel: vtballoon0: <VirtIO Balloon Adapter> on virtio_pci0
Apr 14 14:01:41 trogdor kernel: virtio_pci1: <VirtIO PCI SCSI adapter> port 0xe000-0xe03f mem 0xfea91000-0xfea91fff,0xfe404000-0xfe407fff irq 10 at device 5.0 on pci0
Apr 14 14:01:41 trogdor kernel: vtscsi0: <VirtIO SCSI Adapter> on virtio_pci1
Apr 14 14:01:41 trogdor kernel: virtio_pci2: <VirtIO PCI Network adapter> port 0xe080-0xe09f mem 0xfea92000-0xfea92fff,0xfe408000-0xfe40bfff irq 10 at device 18.0 on pci0
Apr 14 14:01:41 trogdor kernel: vtnet0: <VirtIO Networking Adapter> on virtio_pci2
Apr 14 14:01:41 trogdor kernel: vtnet0: Ethernet address: 02:6e:58:1a:48:34
Apr 14 14:01:41 trogdor kernel: 000.000061 [ 421] vtnet_netmap_attach       max rings 1
Apr 14 14:01:41 trogdor kernel: vtnet0: netmap queues/slots: TX 1/1024, RX 1/1024
Apr 14 14:01:41 trogdor kernel: 000.000062 [ 426] vtnet_netmap_attach       virtio attached txq=1, txd=1024 rxq=1, rxd=1024
Apr 14 14:01:41 trogdor kernel: virtio_pci3: <VirtIO PCI Network adapter> port 0xe0a0-0xe0bf mem 0xfea93000-0xfea93fff,0xfe40c000-0xfe40ffff irq 11 at device 19.0 on pci0
Apr 14 14:01:41 trogdor kernel: vtnet1: <VirtIO Networking Adapter> on virtio_pci3
Apr 14 14:01:41 trogdor kernel: vtnet1: Ethernet address: 1a:b3:6c:0f:3c:61
Apr 14 14:01:41 trogdor kernel: 000.000063 [ 421] vtnet_netmap_attach       max rings 1
Apr 14 14:01:41 trogdor kernel: vtnet1: netmap queues/slots: TX 1/1024, RX 1/1024
Apr 14 14:01:41 trogdor kernel: 000.000064 [ 426] vtnet_netmap_attach       virtio attached txq=1, txd=1024 rxq=1, rxd=1024
Apr 14 14:01:41 trogdor kernel: pcib1: <ACPI PCI-PCI bridge> mem 0xfea94000-0xfea940ff irq 10 at device 30.0 on pci0
Apr 14 14:01:41 trogdor kernel: pci1: <ACPI PCI bus> on pcib1
Apr 14 14:01:41 trogdor kernel: pcib2: <ACPI PCI-PCI bridge> mem 0xfea95000-0xfea950ff irq 11 at device 31.0 on pci0
Apr 14 14:01:41 trogdor kernel: pci2: <ACPI PCI bus> on pcib2
Apr 14 14:01:41 trogdor kernel: acpi_syscontainer0: <System Container> on acpi0
Apr 14 14:01:41 trogdor kernel: acpi_syscontainer1: <System Container> port 0xaf00-0xaf0b on acpi0
Apr 14 14:01:41 trogdor kernel: acpi_syscontainer2: <System Container> port 0xafe0-0xafe3 on acpi0
Apr 14 14:01:41 trogdor kernel: acpi_syscontainer3: <System Container> port 0xae00-0xae13 on acpi0
Apr 14 14:01:41 trogdor kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Apr 14 14:01:41 trogdor kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Apr 14 14:01:41 trogdor kernel: kbd0 at atkbd0
Apr 14 14:01:41 trogdor kernel: atkbd0: [GIANT-LOCKED]
Apr 14 14:01:41 trogdor kernel: psm0: <PS/2 Mouse> irq 12 on atkbdc0
Apr 14 14:01:41 trogdor kernel: psm0: [GIANT-LOCKED]
Apr 14 14:01:41 trogdor kernel: psm0: model IntelliMouse Explorer, device ID 4
Apr 14 14:01:41 trogdor kernel: fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Apr 14 14:01:41 trogdor kernel: fdc0: does not respond
Apr 14 14:01:41 trogdor kernel: device_attach: fdc0 attach returned 6
Apr 14 14:01:41 trogdor kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc97ff,0xeb800-0xeffff on isa0
Apr 14 14:01:41 trogdor kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Apr 14 14:01:41 trogdor kernel: attimer0: <AT timer> at port 0x40 on isa0
Apr 14 14:01:41 trogdor kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Apr 14 14:01:41 trogdor kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Apr 14 14:01:41 trogdor kernel: fdc0: No FDOUT register!
Apr 14 14:01:41 trogdor kernel: ppc0: cannot reserve I/O port range
Apr 14 14:01:41 trogdor kernel: Timecounters tick every 10.000 msec
Apr 14 14:01:41 trogdor kernel: nvme cam probe device init
Apr 14 14:01:41 trogdor kernel: ugen0.1: <Intel UHCI root HUB> at usbus0
Apr 14 14:01:41 trogdor kernel: uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Apr 14 14:01:41 trogdor kernel: uhub0: 2 ports with 2 removable, self powered
Apr 14 14:01:41 trogdor kernel: ugen0.2: <QEMU QEMU USB Tablet> at usbus0
Apr 14 14:01:41 trogdor kernel: uhid0 on uhub0
Apr 14 14:01:41 trogdor kernel: uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 2.00/0.00, addr 2> on usbus0
Apr 14 14:01:41 trogdor kernel: da0 at vtscsi0 bus 0 scbus2 target 0 lun 0
Apr 14 14:01:41 trogdor kernel: da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
Apr 14 14:01:41 trogdor kernel: da0: 300.000MB/s transfers
Apr 14 14:01:41 trogdor kernel: da0: Command Queueing enabled
Apr 14 14:01:41 trogdor kernel: da0: 32768MB (67108864 512 byte sectors)
Apr 14 14:01:41 trogdor kernel: Trying to mount root from ufs:/dev/gptid/3eab9db1-026d-11e8-8563-f9e56acabbc9 [rw]...
Apr 14 14:01:41 trogdor kernel: CPU: Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz (1596.42-MHz K8-class CPU)
Apr 14 14:01:41 trogdor kernel: Origin="GenuineIntel"  Id=0x306d4  Family=0x6  Model=0x3d  Stepping=4
Apr 14 14:01:41 trogdor kernel: Features=0x1f8bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT>
Apr 14 14:01:41 trogdor kernel: Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
Apr 14 14:01:41 trogdor kernel: AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Apr 14 14:01:41 trogdor kernel: AMD Features2=0x121<LAHF,ABM,Prefetch>
Apr 14 14:01:41 trogdor kernel: Structured Extended Features=0x1c07ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP>
Apr 14 14:01:41 trogdor kernel: Structured Extended Features3=0x4000000<IBPB>
Apr 14 14:01:41 trogdor kernel: XSAVE Features=0x1<XSAVEOPT>
Apr 14 14:01:41 trogdor kernel: Hypervisor: Origin = "KVMKVMKVM" 
Apr 14 14:01:41 trogdor kernel: padlock0: No ACE support.
Apr 14 14:01:41 trogdor kernel: aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
Apr 14 14:01:41 trogdor kernel: vtnet0: link state changed to UP
Apr 14 14:01:41 trogdor kernel: vtnet1: link state changed to UP
Apr 14 14:01:41 trogdor kernel: ng0: changing name to 'pppoe0'
Apr 14 14:01:41 trogdor kernel: tun1: changing name to 'ovpns1'
Apr 14 14:01:41 trogdor kernel: ovpns1: link state changed to UP
Apr 14 14:01:41 trogdor kernel: tun2: changing name to 'ovpns2'
Apr 14 14:01:41 trogdor kernel: ovpns2: link state changed to UP
Apr 14 14:01:41 trogdor kernel: pflog0: promiscuous mode enabled
Apr 14 14:01:41 trogdor php-fpm[343]: /rc.start_packages: Restarting/Starting all packages.
Apr 14 14:01:41 trogdor kernel: done.
Apr 14 14:01:41 trogdor ppp: [wan] IPV6CP: SendConfigReq #8
Apr 14 14:01:41 trogdor root: /usr/local/etc/rc.d/dbus: WARNING: failed to start dbus
Apr 14 14:01:42 trogdor php-cgi: notify_monitor.php: Message sent to XX@XX.com OK
Apr 14 14:01:43 trogdor login: login on ttyv0 as root

I'm aware this is a very useless bug report as it stands, given that I haven't pinpointed at all where I think the problem could lie. The only thing I can reliably reproduce is the performance is better when I boot my 2.4.2 virtual instance, vs my 2.4.3 instance. I also noticed the problem straight away after the upgrade.

The only other bit of information that may or may not be useless is that I notice when doing a speedtest the graph on the summary page of pfSense will show WAN hitting 100Mb/s, even though what the LAN side is seeing is greatly reduced. But monitoring pfTop shows no packets being lost.

I have posted a forum topic here where we discussed it a little - https://forum.pfsense.org/index.php?topic=146258.0