Working Cluster in XenServer 7.0 with PV NICs running 2.3.2_1 Upgrade secondary to 2.3.3-DEV then to 2.4.0-DEV Enter CARP Maintenance mode on primary Upgrade primary to 2.3.3-DEV CARP OK Upgrade to 2.4.0-DEV CARP OK in BACKUP state Leave CARP Maintenance mode on primary Nov 4 02:57:27 php-fpm 286 /status_carp.php: The command ‘/sbin/ifconfig xn1 ‘172.25.228.17’/’24’ alias vhid ‘1’’ returned exit code ‘1’, the output was ‘ifconfig: ioctl (SIOCAIFADDR): Protocol not available’ Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn1: 3 Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn1: 3 Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn1: 3 Nov 4 02:57:27 kernel carp: 1@xn1: BACKUP -> INIT (hardware interface up) Nov 4 02:57:27 kernel xn1: promiscuous mode disabled Nov 4 02:57:27 php-fpm 286 /status_carp.php: The command ‘/sbin/ifconfig xn0 ‘172.25.236.1’/’24’ alias vhid ‘2’’ returned exit code ‘1’, the output was ‘ifconfig: ioctl (SIOCAIFADDR): Protocol not available’ Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn0: 3 Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn0: 3 Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn0: 3 Nov 4 02:57:27 kernel carp: 2@xn0: BACKUP -> INIT (hardware interface up) Nov 4 02:57:27 kernel xn0: promiscuous mode disabled Nov 4 02:57:27 php-fpm 286 /status_carp.php: The command ‘/sbin/ifconfig xn2 ‘172.25.237.1’/’24’ alias vhid ‘3’’ returned exit code ‘1’, the output was ‘ifconfig: ioctl (SIOCAIFADDR): Protocol not available’ Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn2: 3 Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn2: 3 Nov 4 02:57:27 kernel ifa_maintain_loopback_route: deletion failed for interface xn2: 3 Nov 4 02:57:27 kernel carp: 3@xn2: BACKUP -> INIT (hardware interface up) Nov 4 02:57:27 kernel xn2: promiscuous mode disabled Nov 4 02:57:27 check_reload_status Syncing firewall Nov 4 02:57:27 check_reload_status Carp backup event Nov 4 02:57:27 check_reload_status Carp backup event Nov 4 02:57:27 check_reload_status Carp backup event Nov 4 02:57:28 php-fpm 285 /rc.filter_synchronize: Beginning XMLRPC sync data to https://172.25.254.2:443/xmlrpc.php. Nov 4 02:57:28 php-fpm 284 /rc.carpbackup: HA cluster member “(172.25.228.17@xn1): (WAN)” has resumed CARP state “BACKUP” for vhid 1 Nov 4 02:57:28 php-fpm 284 /rc.carpbackup: HA cluster member “(172.25.237.1@xn2): (OPT1)” has resumed CARP state “BACKUP” for vhid 3 Nov 4 02:57:28 php-fpm 286 /rc.carpbackup: HA cluster member “(172.25.236.1@xn0): (LAN)” has resumed CARP state “BACKUP” for vhid 2 Nov 4 02:57:28 php-fpm 285 /rc.filter_synchronize: XMLRPC reload data success with https://172.25.254.2:443/xmlrpc.php (pfsense.host_firmware_version). Nov 4 02:57:28 php-fpm 285 /rc.filter_synchronize: XMLRPC versioncheck:15.6 -- 15.6 Nov 4 02:57:28 php-fpm 285 /rc.filter_synchronize: Beginning XMLRPC sync data to https://172.25.254.2:443/xmlrpc.php. Nov 4 02:57:29 php-fpm 285 /rc.filter_synchronize: XMLRPC reload data success with https://172.25.254.2:443/xmlrpc.php (pfsense.restore_config_section). Nov 4 02:57:29 php-fpm 285 /rc.filter_synchronize: Beginning XMLRPC sync data to https://172.25.254.2:443/xmlrpc.php. Nov 4 02:57:32 php-fpm 285 /rc.filter_synchronize: XMLRPC reload data success with https://172.25.254.2:443/xmlrpc.php (pfsense.filter_configure). Secondary still CARP Master Status > CARP blank status for all VIPS on primary No CARP VIPs on interface: xn0: flags=8843 metric 0 mtu 1500 options=3 ether ae:3e:72:48:94:c0 inet6 fe80::ac3e:72ff:fe48:94c0%xn0 prefixlen 64 scopeid 0x5 inet 172.25.236.2 netmask 0xffffff00 broadcast 172.25.236.255 nd6 options=21 media: Ethernet manual status: active xn1: flags=8843 metric 0 mtu 1500 options=3 ether f2:92:fa:6a:32:79 inet6 fe80::f092:faff:fe6a:3279%xn1 prefixlen 64 scopeid 0x6 inet 172.25.228.18 netmask 0xffffff00 broadcast 172.25.228.255 nd6 options=21 media: Ethernet manual status: active xn2: flags=8843 metric 0 mtu 1500 options=3 ether 76:e9:15:72:72:39 inet6 fe80::74e9:15ff:fe72:7239%xn2 prefixlen 64 scopeid 0x7 inet 172.25.237.2 netmask 0xffffff00 broadcast 172.25.237.255 nd6 options=21 media: Ethernet manual status: active xn3: flags=8843 metric 0 mtu 1500 options=3 ether 2a:ce:85:9d:aa:49 inet6 fe80::28ce:85ff:fe9d:aa49%xn3 prefixlen 64 scopeid 0x8 inet 172.25.254.1 netmask 0xfffffffc broadcast 172.25.254.3 nd6 options=21 media: Ethernet manual status: active Reboot Primary Came up and assumed master on all CARP VIPS. Secondary went to all backup. Enter CARP Maintenance mode on Primary. CARP VIPs vanish. No Status in Status > CARP. No CARP VIPs on interfaces, same syslogs as above. Leave CARP Maintenance mode. VIPs reestablished on interfaces, cluster enters normal MASTER/BACKUP status.