Bug #8815
open
IP addresses are removed from interfaces when link is lost and either IPv4 or IPv6 is dynamic
0%
Description
When an interface loses link the IPv4 addresses, including VIPs, can disappear from the interface. If the link is down when the interface settings are applied, the address is present, but plugging in the interface and then removing the link again can make the addresses disappear.
The behavior is not consistent across all configurations and hardware. For example, it appears to happen predictably on igb(4) interfaces but not vmx(4) interfaces, and in some cases re(4) is OK and fails for others.
The main consequence is that anything bound to or relying on those addresses will fail when they disappear. The highest impact is from HA/CARP. The VIPs disappear from the interface, so the OS does not trigger demotion as expected with link loss.
IPv6 addresses, if any exist, are still present on the interfaces after link is lost.
For example, on an SG-5100. LAN is unplugged before the test:
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:90:0b:7a:8a:66
hwaddr 00:90:0b:7a:8a:66
inet 10.23.0.1 netmask 0xffffff00 broadcast 10.23.0.255
inet6 2001:db8:1:ee10:290:bff:fe7a:8a66 prefixlen 60
inet6 fe80::1:1%igb1 prefixlen 64 scopeid 0x2
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
Plug in LAN, then unplug LAN:
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:90:0b:7a:8a:66
hwaddr 00:90:0b:7a:8a:66
inet6 2001:db8:1:ee10:290:bff:fe7a:8a66 prefixlen 60
inet6 fe80::1:1%igb1 prefixlen 64 scopeid 0x2
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
Logs for igb1 during the transition:
Aug 20 11:54:29 missy kernel: igb1: link state changed to UP Aug 20 11:54:29 missy check_reload_status: Linkup starting igb1 Aug 20 11:54:29 missy check_reload_status: Reloading filter Aug 20 11:54:30 missy php-fpm[585]: /rc.linkup: DEVD Ethernet attached event for lan Aug 20 11:54:30 missy php-fpm[585]: /rc.linkup: HOTPLUG: Configuring interface lan Aug 20 11:54:30 missy check_reload_status: Restarting ipsec tunnels Aug 20 11:54:33 missy check_reload_status: Linkup starting igb1 Aug 20 11:54:33 missy kernel: igb1: link state changed to DOWN Aug 20 11:54:34 missy php-fpm[584]: /rc.newwanip: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1534780474] unbound[62993:0] error: bind: address already in use [1534780474] unbound[62993:0] fatal error: could not open ports' Aug 20 11:54:34 missy check_reload_status: updating dyndns lan Aug 20 11:54:34 missy php-fpm[39495]: /rc.linkup: DEVD Ethernet detached event for lan Aug 20 11:54:35 missy check_reload_status: Reloading filter
Updated by Steve Wheeler over 6 years ago
I have replicated this in em(4) and re(4) (on an APU) but not on mvneta(4).
All on the latest 2.4.4 snapshot.
Updated by Anonymous about 6 years ago
- Target version changed from 2.4.4 to 2.4.4-GS
Updated by
Updated by Anonymous about 6 years ago
- Target version changed from 2.4.4-GS to 2.4.4-p1
Updated by Anonymous about 6 years ago
- Target version changed from 2.4.4-p1 to 48
Updated by Anonymous about 4 years ago
- Target version changed from 2.5.0 to CE-Next
Updated by Steve Wheeler over 3 years ago
This appears to be coupled to the IPv6 setting on the interface that is disconnected.
Testing here against 2.5.2, with the LAN set to 'Track Interface' and tracking WAN which is DHCPv6 the IP addresses on the interface are lost when it is disconnected.
With it set to None or as a valid Static IPv6 address all IP addresses are retained.
Updated by Jim Pingle over 3 years ago
- Subject changed from IPv4 addresses disappear from interfaces when link is lost to IP addresses are removed from interfaces when link is lost and either IPv4 or IPv6 is dynamic
- Priority changed from Very High to Normal
- Plus Target Version set to Plus-Next
This appears to be due to an interface having either IPv4 or IPv6 set to a dynamic type (read: Not static, not 'none'). If both are static or one is static and the other set to 'none' this doesn't happen.
In the case where either type of address is dynamic a link loss triggers a different code path which runs interface_bring_down() on the interface which lost link. That function in turn runs ifconfig <name> delete which clears the IPv4 addresses and other functions which have similar effects, the end result being that all addresses are cleared from the interface when it goes down.
For typical cases this is fairly low impact since the addresses come back when the interface regains link. However, if a system is part of an HA cluster using CARP VIPs, this can be a problem. When the CARP VIP is removed, rather than kept in place, it causes problems with demotion and preemption so failover won't function.
Dynamic addresses are not compatible with HA, so HA nodes should not be configured in this way. If IPv6 is not going to be used, then all interfaces should have IPv6 set to "None" rather than leaving DHCPv6 or Track Interface selected.
That said, it might be nice to detect such cases and not be so harsh. For example, maybe when link is lost, only addresses of the dynamic type (IPv4 or IPv6) are cleared when mixing static and dynamic on an interface.
Updating subject and lowering priority to reflect what we've found so far.