Bug #14586
closed
Adding an IP Alias VIP using a unicast CARP VIP as its parent changes the CARP VIP to multicast at the OS level
0%
Description
Adding IP Alias to a unicast CARP VIP results in the CARP VIP being reconfigured to multicast. Reapplying (i.e. saving) the unicast CARP VIP sets it back to unicast, however, whenever the IP Alias is saved or the instance is rebooted, the associated unicast CARP VIP becomse multicast again.
This breaks unicast CARP.
A bug exists in /etc/inc/interfaces.inc function interface_ipalias_configure() in that it does not check for ucast/mcast as interface_carp_configure() does. The resultant command executed (e.g ifconfig mce0 inet 172.20.0.76/26 alias vhid 70) is missing the "peer" statement, this causes a change in the underlying CARP VIP configuration.
The attached patch rectified this by adding a ucast/mcast check and a "peer <address>" or "mcast" statement.
Files
Updated by Jim Pingle over 1 year ago
- Assignee set to Reid Linnemann
- Target version set to 23.09
Updated by Danilo Zrenjanin over 1 year ago
I've tested against:
23.05.1-RELEASE (amd64) built on Wed Jun 28 03:57:27 UTC 2023 FreeBSD 14.0-CURRENT
I couldn't reproduce the reported issue. Making an Alias on top of an existing CARP VIP, which has a unicast option checked, doesn't change the CARP Mode of the parent CARP VIP. Rebooting the firewall didn't change the CARP mode on the parent CARP VIP.
Updated by James George over 1 year ago
Interesting, I'm definitely seeing this on 23.05.1 (just noticed I selected the wrong version in the bug - I'll fix that). I'll check again and gather a bit more info.
Could this be related to the underlying NIC type?
Updated by James George over 1 year ago
I backed out my patch and rebooted. Looking at just LAN:
ifconfig mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: LAN
options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
ether 02:00:17:01:82:50
inet6 fe80::17ff:fe01:8250%mce0 prefixlen 64 scopeid 0x5
inet6 xxxx:xxxx:xxxx:xxxx::11 prefixlen 64
inet6 xxxx:xxxx:xxxx:xxxx::10 prefixlen 64 vhid 80
inet6 xxxx:xxxx:xxxx:xxxx::15 prefixlen 64 vhid 80
inet 172.20.0.71 netmask 0xffffffc0 broadcast 172.20.0.127
inet 172.20.0.70 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
inet 172.20.0.75 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
inet 172.20.0.76 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
carp: MASTER vhid 70 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
carp: MASTER vhid 80 advbase 2 advskew 0
peer 224.0.0.18 peer6 ff02::12
media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Definitely reverted to mcast.
Saving the VIPs again through the UI:
ifconfig mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: LAN
options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
ether 02:00:17:01:82:50
inet6 fe80::17ff:fe01:8250%mce0 prefixlen 64 scopeid 0x5
inet6 xxxx:xxxx:xxxx:xxxx::11 prefixlen 64
inet6 xxxx:xxxx:xxxx:xxxx::15 prefixlen 64 vhid 80
inet6 xxxx:xxxx:xxxx:xxxx::10 prefixlen 64 vhid 80
inet 172.20.0.71 netmask 0xffffffc0 broadcast 172.20.0.127
inet 172.20.0.75 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
inet 172.20.0.76 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
inet 172.20.0.70 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
carp: MASTER vhid 70 advbase 1 advskew 0
peer 172.20.0.72 peer6 ff02::12
carp: MASTER vhid 80 advbase 2 advskew 0
peer 224.0.0.18 peer6 xxxx:xxxx:xxxx:xxxx::12
media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Set to ucast as I'd expect.
Save one of the v4 aliases on VIP (VHID 70):
ifconfig mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: LAN
options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
ether 02:00:17:01:82:50
inet6 fe80::17ff:fe01:8250%mce0 prefixlen 64 scopeid 0x5
inet6 xxxx:xxxx:xxxx:xxxx::11 prefixlen 64
inet6 xxxx:xxxx:xxxx:xxxx::15 prefixlen 64 vhid 80
inet6 xxxx:xxxx:xxxx:xxxx::10 prefixlen 64 vhid 80
inet 172.20.0.71 netmask 0xffffffc0 broadcast 172.20.0.127
inet 172.20.0.76 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
inet 172.20.0.70 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
inet 172.20.0.75 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
carp: MASTER vhid 70 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
carp: MASTER vhid 80 advbase 2 advskew 0
peer 224.0.0.18 peer6 2603:c023:c003:4a01::12
media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
That VIP is back to mcast and the other (VHID 80) is still ucast.
The UI always shows that the VIP is set to unicast, but the NIC is definitely not configured unicast when Aliases are in play.
23.05.1-RELEASE (amd64) built on Wed Jun 28 03:57:27 UTC 2023 FreeBSD 14.0-CURRENT AMD EPYC 7J13 64-Core Processor 6 CPUs : 1 package(s) x 3 core(s) x 2 hardware threads AES-NI CPU Crypto: Yes (active) IPsec-MB Crypto: Yes (active) QAT Crypto: No
These are virtualised hosts, NICs are presented SRIOV/VFIO (Mellanox I believe).
Updated by Danilo Zrenjanin over 1 year ago
- Status changed from New to Confirmed
Ok, that was my bad. I've checked only the GUI status. After checking the ifconfig output, I realized that it reverts the parent CARP VIP to the multicast as initially reported. I'll mark this report as confirmed.
Updated by Reid Linnemann over 1 year ago
I've got a similar patch incoming, and this should be included in the System Patches as well I think.
Updated by Reid Linnemann over 1 year ago
- Status changed from Confirmed to Feedback
Fixed in eab8453f
Updated by Jim Pingle over 1 year ago
- Subject changed from Adding IP Alias to unicast CARP VIP "undoes" unicast reverting to multicast to Adding an IP Alias VIP using a unicast CARP VIP as its parent changes the CARP VIP to multicast at the OS level
Updating subject for release notes.
Updated by James George over 1 year ago
I'm happy to test the fix in my environment if you'd like; I'd just need a diff/patch to apply if the official fix is materially different to the one I attached.
Updated by Lev Prokofev over 1 year ago
Tested on Dev build
23.09-DEVELOPMENT (amd64)
built on Wed Aug 09 06:05:37 UTC 2023
FreeBSD 14.0-CURRENT
Adding/redacting an Alias IP doesn't change the CARP type.
igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:08:a2:0b:c4:58
inet 10.150.100.10 netmask 0xffffff00 broadcast 10.150.100.255
inet 10.150.100.25 netmask 0xffffffff broadcast 10.150.100.25 vhid 1
inet 10.150.100.20 netmask 0xffffffff broadcast 10.150.100.20 vhid 2
inet 10.150.100.23 netmask 0xffffffff broadcast 10.150.100.23 vhid 1
inet 10.150.100.13 netmask 0xffffffff broadcast 10.150.100.13 vhid 2
inet6 fe80::208:a2ff:fe0b:c458%igb0 prefixlen 64 scopeid 0x1
carp: MASTER vhid 1 advbase 1 advskew 0
peer 10.150.100.2 peer6 ff02::12
carp: MASTER vhid 2 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Updated by Reid Linnemann over 1 year ago
- File 14587_interfaces.inc.diff added
James George wrote in #note-9:
I'm happy to test the fix in my environment if you'd like; I'd just need a diff/patch to apply if the official fix is materially different to the one I attached.
Be my guest! Patch is attached.
Updated by James George over 1 year ago
- File interfaces_inc_new.patch interfaces_inc_new.patch added
Thanks Reid.
Unfortunately, this seems to only be a partial fix (for me at least) - it does not work at bootup. I applied the patch to my two 23.05.1 instance and rebooted the primary. I then see this after boot up:
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: LAN
options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
ether 02:00:17:01:82:50
inet6 fe80::17ff:fe01:8250%mce0 prefixlen 64 scopeid 0x5
inet6 xxxx:xxxx:xxxx:xxxx::11 prefixlen 64
inet6 xxxx:xxxx:xxxx:xxxx::10 prefixlen 64 vhid 80
inet6 xxxx:xxxx:xxxx:xxxx::15 prefixlen 64 vhid 80
inet 172.20.0.71 netmask 0xffffffc0 broadcast 172.20.0.127
inet 172.20.0.70 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
inet 172.20.0.75 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
inet 172.20.0.76 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
carp: MASTER vhid 70 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
carp: MASTER vhid 80 advbase 2 advskew 0
peer 224.0.0.18 peer6 ff02::12
media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mce1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: WAN
options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
ether 02:00:17:01:ee:2b
inet6 fe80::17ff:fe01:ee2b%mce1 prefixlen 64 scopeid 0x6
inet 172.20.0.11 netmask 0xffffffc0 broadcast 172.20.0.63
inet 172.20.0.10 netmask 0xffffffc0 broadcast 172.20.0.63 vhid 10
carp: MASTER vhid 10 advbase 1 advskew 0
peer 172.20.0.12 peer6 ff02::12
media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
So, WAN is ok, but LAN is back at being mcast.
The issue seems to be $vip['carp_mode'] 'ucast' isn't "correct" during boot, where as $carpvip['carp_mode'] 'ucast' is (I've not dug further into the code to understand why).
Modifying your patch to (attached as well):
--- interfaces.inc.orig 2023-08-15 10:07:01.006137000 +0800
+++ interfaces.inc 2023-08-15 10:38:14.251255000 +0800
@@ -3054,9 +3054,14 @@
$carpvip = get_configured_vip($vip['interface']);
$iface = $carpvip['interface'];
$vhid = "vhid {$carpvip['vhid']}";
+ if ($af == 'inet') {
+ $mode = ($carpvip['carp_mode'] == 'ucast' ? "peer ". escapeshellarg($carpvip['carp_peer']) : " mcast");
+ } else {
+ $mode = ($carpvip['carp_mode'] == 'ucast' ? "peer6 ". escapeshellarg($carpvip['carp_peer']) : " mcast6");
+ }
}
- mwexec("/sbin/ifconfig " . escapeshellarg($realif) ." {$af} ". escapeshellarg($vip['subnet']) ."/" . escapeshellarg($vip['subnet_bits']) . " alias {$gateway} {$vhid}");
- unset($iface, $af, $realif, $carpvip, $vhid, $gateway);
+ mwexec("/sbin/ifconfig " . escapeshellarg($realif) ." {$af} ". escapeshellarg($vip['subnet']) ."/" . escapeshellarg($vip['subnet_bits']) . " alias {$gateway} {$vhid} {$mode}");
+ unset($iface, $af, $realif, $carpvip, $vhid, $gateway, $mode);
}
function interface_carp_configure(&$vip, $ipalias_reload = false) {
Things now work as expected for me.
Updated by Reid Linnemann over 1 year ago
Oh shoot, I apologize. I created the patch from a previous aborted MR, which I had closed before I saw and corrected that copy/paste error. I'll remove the bad patch for posterity. Your current patch reflects the actual change that went into the codebase.
Updated by Reid Linnemann over 1 year ago
- File deleted (
14587_interfaces.inc.diff)
Updated by James George over 1 year ago
Ok, cool. Thanks for letting me know. I'll await 23.09. :)