pfSense » pfSense Plus

forgot to add: without OpenVPN running on VIP or even with OpenVPN runnning on WAN, there is no problem with TCP streams during HA failover test.

Verified. Running OpenVPN server bound to Localhost and port forwarding an IP Alias/CARP VIP to it looks like a reasonable workaround for now.

The state vanishes from the cluster member being failed to.

It remains active on the other node so there’s some sort of breakdown in pfsync there too.

The state will hang around on the original node since the app will die and it’ll just sit there in ESTABLISHED:ESTABLISHED for 24 hours.

At least in this test case. VLC gives up pretty quickly.

So 1. The state shouldn’t be killed. 2. if it is killed it should be deleted on the original node by pfsync.

Additional notes while working with cjl:
Commenting out the line /sbin/pfctl -i $1 -Fs in /usr/local/sbin/ovpn-linkdown "fixes" the issue. However, the script should not be getting called unless the service has already started and stopped; in the case of a failover to the secondary, the service should not be running while in BACKUP status, hence the script shouldn't be executed.

Hi there, I think I am seeing the same issue (on 23.05). I also do have OpenVPN on CARP IPs as of now (though openvpn might not be the only factor causing this). I'll add as much detail here as possible, let me know what you think.

My setup is like this:

client(192.168.11.43) -- (192.168.11.2 on vtnet2.511) pfsense1 (10.7.200.2 on vtnet2.192) -- target(10.7.200.12)
                       | (192.168.11.3 on vtnet2.511) pfsense2 (10.7.200.3 on vtnet2.192) |

So a pretty normal HA-Cluster. Client & Target are both in local networks, OpenVPN is not involved in the connections but is running. During a failover I see no errors anywhere but an SSH session from client to target hangs. Associated with that I see the following blocks in the filter log:

Jun  9 14:28:57 pfSense2 filterlog[94533]: 4,,,1000000103,vtnet2.511,match,block,in,4,0x48,,64,41324,0,DF,6,tcp,124,192.168.11.43,10.7.200.12,51568,22,72,PA,1544295413:1544295485,73422640,501,,nop;nop;TS
Jun  9 14:30:07 pfSense2 filterlog[94533]: 4,,,1000000103,vtnet2.192,match,block,in,4,0x48,,64,44921,0,DF,6,tcp,104,10.7.200.12,192.168.11.43,22,46976,52,PA,716626969:716627021,2743249057,501,,nop;nop;TS
Jun  9 14:30:08 pfSense2 filterlog[94533]: 4,,,1000000103,vtnet2.192,match,block,in,4,0x48,,64,44922,0,DF,6,tcp,104,10.7.200.12,192.168.11.43,22,46976,52,PA,716626969:716627021,2743249057,501,,nop;nop;TS
Jun  9 14:30:08 pfSense2 filterlog[94533]: 4,,,1000000103,vtnet2.192,match,block,in,4,0x48,,64,44923,0,DF,6,tcp,104,10.7.200.12,192.168.11.43,22,46976,52,PA,716626969:716627021,2743249057,501,,nop;nop;TS
Jun  9 14:30:08 pfSense2 filterlog[94533]: 4,,,1000000103,vtnet2.192,match,block,in,4,0x48,,64,44924,0,DF,6,tcp,104,10.7.200.12,192.168.11.43,22,46976,52,PA,716626969:716627021,2743249057,501,,nop;nop;TS
Jun  9 14:30:09 pfSense2 filterlog[94533]: 4,,,1000000103,vtnet2.192,match,block,in,4,0x48,,64,44925,0,DF,6,tcp,104,10.7.200.12,192.168.11.43,22,46976,52,PA,716626969:716627021,2743249057,501,,nop;nop;TS
Jun  9 14:30:11 pfSense2 filterlog[94533]: 4,,,1000000103,vtnet2.192,match,block,in,4,0x48,,64,44926,0,DF,6,tcp,104,10.7.200.12,192.168.11.43,22,46976,52,PA,716626969:716627021,2743249057,501,,nop;nop;TS

This indicates to me that pfSense2 doesn't have the required states or is not able to match them up properly.

On the other hand, grepping for those states properly shows them on both firewalls:

pfctl -s states | grep '10.7.200.12:22'
all tcp 10.7.200.12:22 <- 10.7.22.120:51714       ESTABLISHED:ESTABLISHED
all tcp 10.7.22.120:51714 -> 10.7.200.12:22       ESTABLISHED:ESTABLISHED

Marcos M wrote in #note-5:

Additional notes while working with cjl:
Commenting out the line /sbin/pfctl -i $1 -Fs in /usr/local/sbin/ovpn-linkdown "fixes" the issue. However, the script should not be getting called unless the service has already started and stopped; in the case of a failover to the secondary, the service should not be running while in BACKUP status, hence the script shouldn't be executed.

But this would executed on the master during failover, no? And whatever that does to states would get synced to the secondary?

As an extra note: If I fail back to the primary again, the SSH connections resumes working…

I am able to reproduce the issue and I can also confirm that the issue is gone if I comment out /sbin/pfctl -i $1 -Fs. I can also confirm that the states actually vanish from the second firewall during the failover (so something must be actively deleting them?). Interestingly enough if I manually execute /sbin/pfctl -i ovpns1 -Fs it says 0 states cleared and nothing happens on the secondary firewall :/ I am somewhat out of ideas.

Debugging even further this seems to be timing sensitive. If I run pfctl -i ovpns1 -Fs && pfSctl -c 'filter reload all' on the primary, then for a short time the state is missing on the standby. So maybe between the carp events, filter reload and flushing stuff on ovpns1 something weird happens. Is there a way to monitor state tables changes?

Potentially related to https://redmine.pfsense.org/issues/11556

I don't think those two are related.

Same issue on pfsense 2.7,
I confirm that

commenting out the line /sbin/pfctl -i $1 -Fs