pfSense

I am using the AMD64 builds of pfSense 2.0 RC3. I have the same build running on an dedicated hardware and in a VM with CARP configured on both. The dedicated hardware was installed with a build of AMD64 RC2. When I setup CARP, I updated the dedicated hardware to the build used in the VM and installed the VM using pfSense-2.0-RC3-amd64-20110708-1843.iso. I have updated four times since the install. I have had the same issue on all 5 of the builds I have used with the CARP setup. 2.0-RC3-amd64-20110708-1843, 2.0-RC3 Built On: Sun Jul 24 04:39:44 EDT 2011 and 2.0-RC3 (amd64)built on Fri Jul 29 22:14:50 EDT 2011 are builds I can name that have been tested with this setup. 2.0-RC3 (amd64)built on Fri Jul 29 22:14:50 EDT 2011 is the latest I have installed.

1) If I reboot the primary pfSense device (dedicated hardware) all interfaces fail-over to the pfSense VM and with-in 20 seconds my connections are active through the backup pfSense device (VM). When the primary pfSense device is back up, the connections fail-over to it and with-in 20 seconds my connections are active through the primary pfSence device.

This is a success and indicates that my configuration is correct. However there is more.

2) If I physically disconnect the cable for the WAN uplink I see the following on the CARP Status pages on both devices.

Primary: WAN status is init and the status of the other interfaces is backup.
Secondary: WAN status is master and the status of the other interfaces is backup.

Well the WAN interface does fail-over, but the other interfaces do not, so I get nothing useful.

I get the same if I physically disconnect one of the other interfaces. In this case, that interface cannot communicate with the other pfSense interfaces, but the other interfaces work as expected.

3) If I take the WAN interface down via software, with ifconfig em5 down, I get the same result with the current build, but there are minor status differences in previous builds.

Primary: WAN status is backup and the status of the other interfaces is master.
Secondary: WAN status is master and the status of the other interfaces is backup.

I get the same results if I take down an interface other than WAN, except the interface that is down fails over and the others do not.

The bottom line is if all interfaces go down, then it works, but if not then I get nothing useful.

From the BSD docs for CARP it seems that all the Virtual IPs that should fail-over together should have the same VHID group. The pfSense GUI does not allow me to create multiple CARP interfaces using the same VHID group. This indicates that either pfsense uses another method to fail-over all interfaces when one is offline or it is not intended to work this way or there is a bug.

4) This brings up a question. Is pfSense supposed to be able to detect flaky connections or dead routes and fail-over? I have included some examples below.

4a) If the WAN connection on the primary device starts getting errors and or collisions, will pfSense fail-over until the errors and/or collisions stop?

4b) If the WAN connection on the primary device is switching between up and down states, will pfSense fail-over until the connection became more stable?

4c) If primary device cannot reach the WAN gateway, will pfSense fail-over until it can reach the WAN gateway?