pfSense

Upgraded 2.4.4-p3 to snapshot 2020-03-07 0136

I have 3 VMX connected and all seems to be working. No issues.

I appear to have made a typo. It should be more than 3 adapters. I checked ran into it again a week or two ago, it was the same for E1000E also.

I have run into this bug again with a VM and 2.4.5. I am unable to add a second interface at this point, they show up in messages as attached, then detached. I have preserved the VM and can supply credentials as necessary.

The VM was created with 5 VMXNET3 adapters successfully, but the ordering was wrong, so I removed 3. I then removed LAN from assigned interfaces in pfSense GUI and which point it dropped from the VM, any attempts to re-add any new NICs show attach/detach, reboot doesn't re-initialize any of the added nics.

I believe this to be a bug in the vSphere HOST Web GUI. I have run into very similar problems with other hardware configuration steps and have had to remove the VM from inventory, delete the VMX and re-create the VM.

I will re-test at next opportunity.

I've diff'd the VMX files between creating a VM with 2 nics (which works) and creating a VM with 1 NIC and attempting to add a second:

Created with 2 nics:

ethernet0.addressType = "generated" 
ethernet0.generatedAddress = "00:0c:29:c8:07:35" 
ethernet0.generatedAddressOffset = "0" 
ethernet0.networkName = "MGMT - Production" 
ethernet0.pciSlotNumber = "192" 
ethernet0.present = "TRUE" 
ethernet0.uptCompatibility = "TRUE" 
ethernet0.virtualDev = "vmxnet3" 
ethernet0.wakeOnPcktRcv = "FALSE" 
ethernet1.addressType = "generated" 
ethernet1.generatedAddress = "00:0c:29:c8:07:3f" 
ethernet1.generatedAddressOffset = "10" 
ethernet1.networkName = "pfSense sync" 
ethernet1.pciSlotNumber = "224" 
ethernet1.present = "TRUE" 

ethernet1.uptCompatibility = "TRUE" 
ethernet1.virtualDev = "vmxnet3" 
ethernet1.wakeOnPcktRcv = "FALSE" 

Created with 1 nic, add 1 nic:

ethernet0.addressType = "generated" 
ethernet0.generatedAddress = "00:0c:29:93:35:d1" 
ethernet0.generatedAddressOffset = "0" 
ethernet0.networkName = "MGMT - Production" 
ethernet0.pciSlotNumber = "192" 
ethernet0.present = "TRUE" 
ethernet0.uptCompatibility = "TRUE" 
ethernet0.virtualDev = "vmxnet3" 
ethernet0.wakeOnPcktRcv = "FALSE" 
ethernet1.addressType = "generated" 

ethernet1.generatedAddressOffset = "10" 
ethernet1.networkName = "MGMT - Production" 
ethernet1.pciSlotNumber = "-1" 
ethernet1.present = "TRUE" 
ethernet1.startConnected = "FALSE" 
ethernet1.uptCompatibility = "TRUE" 
ethernet1.virtualDev = "vmxnet3" 
ethernet1.wakeOnPcktRcv = "FALSE" 

The PCI slot number is -1 whether the system is running or not. In the running-add nic case, I had the pfSense installer running, and saw the same message as my original post, pci6: detached.

With some more investigation I've found the following:

Created a new single vmxnet3 nic VM.

ethernet0.virtualDev = "vmxnet3" 
ethernet0.networkName = "MGMT - Production" 
ethernet0.addressType = "generated" 
ethernet0.wakeOnPcktRcv = "FALSE" 
ethernet0.uptCompatibility = "TRUE" 
ethernet0.present = "TRUE" 
ethernet0.pciSlotNumber = "192" 
ethernet0.generatedAddress = "00:0c:29:67:40:61" 
ethernet0.generatedAddressOffset = "0" 

ethernet0.virtualDev = "vmxnet3" 
ethernet0.networkName = "MGMT - Production" 
ethernet0.addressType = "generated" 
ethernet0.wakeOnPcktRcv = "FALSE" 
ethernet0.uptCompatibility = "TRUE" 
ethernet0.present = "TRUE" 
ethernet0.pciSlotNumber = "192" 
ethernet0.generatedAddress = "00:0c:29:67:40:61" 
ethernet0.generatedAddressOffset = "0" 
ethernet1.pciSlotNumber = "-1" 
ethernet1.virtualDev = "vmxnet3" 
ethernet1.networkName = "MGMT - Production" 
ethernet1.addressType = "generated" 
ethernet1.uptCompatibility = "TRUE" 
ethernet1.present = "FALSE" 
ethernet1.generatedAddress = "00:0c:29:67:40:6b" 
ethernet1.generatedAddressOffset = "10" 

Power off VM, add add another vmxnet3 nic:

ethernet0.virtualDev = "vmxnet3" 
ethernet0.networkName = "MGMT - Production" 
ethernet0.addressType = "generated" 
ethernet0.wakeOnPcktRcv = "FALSE" 
ethernet0.uptCompatibility = "TRUE" 
ethernet0.present = "TRUE" 
ethernet0.generatedAddress = "00:0c:29:67:40:61" 
ethernet1.virtualDev = "vmxnet3" 
ethernet1.networkName = "MGMT - Production" 
ethernet1.addressType = "generated" 
ethernet1.uptCompatibility = "TRUE" 
ethernet1.present = "TRUE" 
ethernet1.wakeOnPcktRcv = "FALSE" 
ethernet0.pciSlotNumber = "192" 
ethernet0.generatedAddressOffset = "0" 
ethernet1.pciSlotNumber = "-1" 
ethernet1.generatedAddressOffset = "10" 

Power on VM:

ethernet0.virtualDev = "vmxnet3" 
ethernet0.networkName = "MGMT - Production" 
ethernet0.addressType = "generated" 
ethernet0.wakeOnPcktRcv = "FALSE" 
ethernet0.uptCompatibility = "TRUE" 
ethernet0.present = "TRUE" 
ethernet0.generatedAddress = "00:0c:29:67:40:61" 
ethernet1.virtualDev = "vmxnet3" 
ethernet1.networkName = "MGMT - Production" 
ethernet1.addressType = "generated" 
ethernet1.uptCompatibility = "TRUE" 
ethernet1.present = "TRUE" 
ethernet1.wakeOnPcktRcv = "FALSE" 
ethernet0.pciSlotNumber = "192" 
ethernet0.generatedAddressOffset = "0" 
ethernet1.pciSlotNumber = "224" 
ethernet1.generatedAddressOffset = "10" 
ethernet1.generatedAddress = "00:0c:29:67:40:6b" 

It seems at this point that adding NICs to running VMs is non-functional, in no combination of tests did a hot-add work, however powered-off VMs were able to add NICs in the above situations. In some cases, if a strict order of operations wasn't followed there ended up being situations when both hot-add and cold-add of additional nics would fail, with multiple orphaned ethernetX. entries in the vmx config.

It seems at this point that this is not clearly a bug with the vsphere web GUI, but possibly a bug relating the interaction between the host and VM. In all cases OpenVM tools is not installed, nor is an operating system, this was all repeated using the pfSense 2.4.5-p1 CE ISO installer and dropping to the rescue shell.