Project

General

Profile

Actions

Bug #14586

closed

Adding an IP Alias VIP using a unicast CARP VIP as its parent changes the CARP VIP to multicast at the OS level

Added by James George about 1 year ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Category:
CARP
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Release Notes:
Default
Affected Plus Version:
21.05.1
Affected Architecture:

Description

Adding IP Alias to a unicast CARP VIP results in the CARP VIP being reconfigured to multicast. Reapplying (i.e. saving) the unicast CARP VIP sets it back to unicast, however, whenever the IP Alias is saved or the instance is rebooted, the associated unicast CARP VIP becomse multicast again.

This breaks unicast CARP.

A bug exists in /etc/inc/interfaces.inc function interface_ipalias_configure() in that it does not check for ucast/mcast as interface_carp_configure() does. The resultant command executed (e.g ifconfig mce0 inet 172.20.0.76/26 alias vhid 70) is missing the "peer" statement, this causes a change in the underlying CARP VIP configuration.

The attached patch rectified this by adding a ucast/mcast check and a "peer <address>" or "mcast" statement.


Files

interfaces.inc.patch (1.23 KB) interfaces.inc.patch Patch to correct interface_ipalias_configure() in interfaces.inc James George, 07/18/2023 05:30 AM
interfaces_inc_new.patch (1.05 KB) interfaces_inc_new.patch James George, 08/15/2023 03:16 AM
Actions #1

Updated by Jim Pingle about 1 year ago

  • Assignee set to Reid Linnemann
  • Target version set to 23.09
Actions #2

Updated by Danilo Zrenjanin 12 months ago

I've tested against:

23.05.1-RELEASE (amd64)
built on Wed Jun 28 03:57:27 UTC 2023
FreeBSD 14.0-CURRENT

I couldn't reproduce the reported issue. Making an Alias on top of an existing CARP VIP, which has a unicast option checked, doesn't change the CARP Mode of the parent CARP VIP. Rebooting the firewall didn't change the CARP mode on the parent CARP VIP.

Actions #3

Updated by James George 12 months ago

Interesting, I'm definitely seeing this on 23.05.1 (just noticed I selected the wrong version in the bug - I'll fix that). I'll check again and gather a bit more info.

Could this be related to the underlying NIC type?

Actions #4

Updated by James George 12 months ago

I backed out my patch and rebooted. Looking at just LAN:

ifconfig mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN
    options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
    ether 02:00:17:01:82:50
    inet6 fe80::17ff:fe01:8250%mce0 prefixlen 64 scopeid 0x5
    inet6 xxxx:xxxx:xxxx:xxxx::11 prefixlen 64
    inet6 xxxx:xxxx:xxxx:xxxx::10 prefixlen 64 vhid 80
    inet6 xxxx:xxxx:xxxx:xxxx::15 prefixlen 64 vhid 80
    inet 172.20.0.71 netmask 0xffffffc0 broadcast 172.20.0.127
    inet 172.20.0.70 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    inet 172.20.0.75 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    inet 172.20.0.76 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    carp: MASTER vhid 70 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    carp: MASTER vhid 80 advbase 2 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Definitely reverted to mcast.

Saving the VIPs again through the UI:

ifconfig mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN
    options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
    ether 02:00:17:01:82:50
    inet6 fe80::17ff:fe01:8250%mce0 prefixlen 64 scopeid 0x5
    inet6 xxxx:xxxx:xxxx:xxxx::11 prefixlen 64
    inet6 xxxx:xxxx:xxxx:xxxx::15 prefixlen 64 vhid 80
    inet6 xxxx:xxxx:xxxx:xxxx::10 prefixlen 64 vhid 80
    inet 172.20.0.71 netmask 0xffffffc0 broadcast 172.20.0.127
    inet 172.20.0.75 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    inet 172.20.0.76 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    inet 172.20.0.70 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    carp: MASTER vhid 70 advbase 1 advskew 0
          peer 172.20.0.72 peer6 ff02::12
    carp: MASTER vhid 80 advbase 2 advskew 0
          peer 224.0.0.18 peer6 xxxx:xxxx:xxxx:xxxx::12
    media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Set to ucast as I'd expect.

Save one of the v4 aliases on VIP (VHID 70):

ifconfig mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN
    options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
    ether 02:00:17:01:82:50
    inet6 fe80::17ff:fe01:8250%mce0 prefixlen 64 scopeid 0x5
    inet6 xxxx:xxxx:xxxx:xxxx::11 prefixlen 64
    inet6 xxxx:xxxx:xxxx:xxxx::15 prefixlen 64 vhid 80
    inet6 xxxx:xxxx:xxxx:xxxx::10 prefixlen 64 vhid 80
    inet 172.20.0.71 netmask 0xffffffc0 broadcast 172.20.0.127
    inet 172.20.0.76 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    inet 172.20.0.70 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    inet 172.20.0.75 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    carp: MASTER vhid 70 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    carp: MASTER vhid 80 advbase 2 advskew 0
          peer 224.0.0.18 peer6 2603:c023:c003:4a01::12
    media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

That VIP is back to mcast and the other (VHID 80) is still ucast.

The UI always shows that the VIP is set to unicast, but the NIC is definitely not configured unicast when Aliases are in play.

23.05.1-RELEASE (amd64)
built on Wed Jun 28 03:57:27 UTC 2023
FreeBSD 14.0-CURRENT

AMD EPYC 7J13 64-Core Processor
6 CPUs : 1 package(s) x 3 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
IPsec-MB Crypto: Yes (active)
QAT Crypto: No 

These are virtualised hosts, NICs are presented SRIOV/VFIO (Mellanox I believe).

Actions #5

Updated by Danilo Zrenjanin 12 months ago

  • Status changed from New to Confirmed

Ok, that was my bad. I've checked only the GUI status. After checking the ifconfig output, I realized that it reverts the parent CARP VIP to the multicast as initially reported. I'll mark this report as confirmed.

Actions #6

Updated by Reid Linnemann 12 months ago

I've got a similar patch incoming, and this should be included in the System Patches as well I think.

Actions #7

Updated by Reid Linnemann 12 months ago

  • Status changed from Confirmed to Feedback

Fixed in eab8453f

Actions #8

Updated by Jim Pingle 12 months ago

  • Subject changed from Adding IP Alias to unicast CARP VIP "undoes" unicast reverting to multicast to Adding an IP Alias VIP using a unicast CARP VIP as its parent changes the CARP VIP to multicast at the OS level

Updating subject for release notes.

Actions #9

Updated by James George 12 months ago

I'm happy to test the fix in my environment if you'd like; I'd just need a diff/patch to apply if the official fix is materially different to the one I attached.

Actions #10

Updated by Lev Prokofev 12 months ago

Tested on Dev build

23.09-DEVELOPMENT (amd64)
built on Wed Aug 09 06:05:37 UTC 2023
FreeBSD 14.0-CURRENT

Adding/redacting an Alias IP doesn't change the CARP type.

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
    ether 00:08:a2:0b:c4:58
    inet 10.150.100.10 netmask 0xffffff00 broadcast 10.150.100.255
    inet 10.150.100.25 netmask 0xffffffff broadcast 10.150.100.25 vhid 1
    inet 10.150.100.20 netmask 0xffffffff broadcast 10.150.100.20 vhid 2
    inet 10.150.100.23 netmask 0xffffffff broadcast 10.150.100.23 vhid 1
    inet 10.150.100.13 netmask 0xffffffff broadcast 10.150.100.13 vhid 2
    inet6 fe80::208:a2ff:fe0b:c458%igb0 prefixlen 64 scopeid 0x1
    carp: MASTER vhid 1 advbase 1 advskew 0
          peer 10.150.100.2 peer6 ff02::12
    carp: MASTER vhid 2 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Actions #11

Updated by Jim Pingle 11 months ago

  • Status changed from Feedback to Resolved
Actions #12

Updated by Reid Linnemann 11 months ago

  • File 14587_interfaces.inc.diff added

James George wrote in #note-9:

I'm happy to test the fix in my environment if you'd like; I'd just need a diff/patch to apply if the official fix is materially different to the one I attached.

Be my guest! Patch is attached.

Actions #13

Updated by James George 11 months ago

Thanks Reid.

Unfortunately, this seems to only be a partial fix (for me at least) - it does not work at bootup. I applied the patch to my two 23.05.1 instance and rebooted the primary. I then see this after boot up:

mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN
    options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
    ether 02:00:17:01:82:50
    inet6 fe80::17ff:fe01:8250%mce0 prefixlen 64 scopeid 0x5
    inet6 xxxx:xxxx:xxxx:xxxx::11 prefixlen 64
    inet6 xxxx:xxxx:xxxx:xxxx::10 prefixlen 64 vhid 80
    inet6 xxxx:xxxx:xxxx:xxxx::15 prefixlen 64 vhid 80
    inet 172.20.0.71 netmask 0xffffffc0 broadcast 172.20.0.127
    inet 172.20.0.70 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    inet 172.20.0.75 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    inet 172.20.0.76 netmask 0xffffffc0 broadcast 172.20.0.127 vhid 70
    carp: MASTER vhid 70 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    carp: MASTER vhid 80 advbase 2 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mce1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: WAN
    options RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,TXRTLMT,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT,RXTLS6
    ether 02:00:17:01:ee:2b
    inet6 fe80::17ff:fe01:ee2b%mce1 prefixlen 64 scopeid 0x6
    inet 172.20.0.11 netmask 0xffffffc0 broadcast 172.20.0.63
    inet 172.20.0.10 netmask 0xffffffc0 broadcast 172.20.0.63 vhid 10
    carp: MASTER vhid 10 advbase 1 advskew 0
          peer 172.20.0.12 peer6 ff02::12
    media: Ethernet 50GBase-KR2 <full-duplex,rxpause,txpause>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

So, WAN is ok, but LAN is back at being mcast.

The issue seems to be $vip['carp_mode'] 'ucast' isn't "correct" during boot, where as $carpvip['carp_mode'] 'ucast' is (I've not dug further into the code to understand why).

Modifying your patch to (attached as well):

--- interfaces.inc.orig    2023-08-15 10:07:01.006137000 +0800
+++ interfaces.inc    2023-08-15 10:38:14.251255000 +0800
@@ -3054,9 +3054,14 @@
         $carpvip = get_configured_vip($vip['interface']);
         $iface = $carpvip['interface'];
         $vhid = "vhid {$carpvip['vhid']}";
+        if ($af == 'inet') {
+            $mode = ($carpvip['carp_mode'] == 'ucast' ? "peer ". escapeshellarg($carpvip['carp_peer']) : " mcast");
+        } else {
+            $mode = ($carpvip['carp_mode'] == 'ucast' ? "peer6 ". escapeshellarg($carpvip['carp_peer']) : " mcast6");
+        }
     }
-    mwexec("/sbin/ifconfig " . escapeshellarg($realif) ." {$af} ". escapeshellarg($vip['subnet']) ."/" . escapeshellarg($vip['subnet_bits']) . " alias {$gateway} {$vhid}");
-    unset($iface, $af, $realif, $carpvip, $vhid, $gateway);
+    mwexec("/sbin/ifconfig " . escapeshellarg($realif) ." {$af} ". escapeshellarg($vip['subnet']) ."/" . escapeshellarg($vip['subnet_bits']) . " alias {$gateway} {$vhid} {$mode}");
+    unset($iface, $af, $realif, $carpvip, $vhid, $gateway, $mode);
 }

 function interface_carp_configure(&$vip, $ipalias_reload = false) {

Things now work as expected for me.

Actions #14

Updated by Reid Linnemann 11 months ago

Oh shoot, I apologize. I created the patch from a previous aborted MR, which I had closed before I saw and corrected that copy/paste error. I'll remove the bad patch for posterity. Your current patch reflects the actual change that went into the codebase.

Actions #15

Updated by Reid Linnemann 11 months ago

  • File deleted (14587_interfaces.inc.diff)
Actions #16

Updated by James George 11 months ago

Ok, cool. Thanks for letting me know. I'll await 23.09. :)

Actions

Also available in: Atom PDF