pfSense

Strange one here:

Testing Load Balancing it seems there is an issue with weights. Given that there are two WANs each with one gateway, if one gateway has a weight of 1, and another gateway has a weight higher than 1 (2, 3, etc), only one of the gateways is used.

Example cases:
WAN=1, WAN2=1: OK

pass in quick on vmx1 route-to { (vmx3 203.0.113.1), (vmx0 198.51.100.1) } round-robin inet from 10.3.0.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"

WAN1=1, WAN2=2: Only uses one gateway (203.0.113.1):

pass in quick on vmx1 route-to { (vmx3 203.0.113.1), (vmx3 203.0.113.1), (vmx0 198.51.100.1) } round-robin inet from 10.3.0.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"

WAN1=2, WAN2=1: Only uses one gateway (203.0.113.1):

pass in quick on vmx1 route-to { (vmx3 203.0.113.1), (vmx0 198.51.100.1), (vmx0 198.51.100.1) } round-robin inet from 10.3.0.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"

If both weights are > 1, then it is OK:
WAN1=2, WAN2=2: OK

pass in quick on vmx1 route-to { (vmx3 203.0.113.1), (vmx3 203.0.113.1), (vmx0 198.51.100.1), (vmx0 198.51.100.1) } round-robin inet from 10.3.0.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"

WAN1=3, WAN2=2: OK

pass in quick on vmx1 route-to { (vmx3 203.0.113.1), (vmx3 203.0.113.1), (vmx0 198.51.100.1), (vmx0 198.51.100.1), (vmx0 198.51.100.1) } round-robin inet from 10.3.0.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"

Also of note, when the weights differ, even though the gateways have a specific order with repetition in the rule, pf seems to still flip back and forth, though the general ratio of the weights is respected. For example with WAN1=3, WAN2=2:

$ for i in {1..10}; do curl http://iptest.example.com/ip.php; done
Your IP Address is: 198.51.100.3
Your IP Address is: 203.0.113.3
Your IP Address is: 198.51.100.3
Your IP Address is: 198.51.100.3
Your IP Address is: 203.0.113.3
Your IP Address is: 198.51.100.3
Your IP Address is: 203.0.113.3
Your IP Address is: 198.51.100.3
Your IP Address is: 198.51.100.3
Your IP Address is: 203.0.113.3

If it turns out to be a deeper issue in pf, the easy fix would be to automatically multiply all weights by 2x if one is 1 and the other is >1.