pfSense

I have done this for pppoe/pptp/l2tp interfaces through pfctl -b in 2.0.
For the other cases some more code analysis is needed to find when the interface fails.

Though its just a matter of writing the code than how-to analysis.

AFAIK the patch is present in pf(4) for 7.2 sources too.

I looked through the tree last night and I did not see pfctl -b in use anywhere. Can someone point me to where its currently being used?

I don't think it is being used, it needs to be tested to ensure it works, then added to run appropriately when a WAN fails.

I already told:
"I have done this for pppoe/pptp/l2tp interfaces through pfctl -b in 2.0."

Look at the link down scripts in /usr/local/[s]bin

That's good Ermal, that fixes a different problem with the same cause where upon PPPoE reconnect (primarily in countries where they force a daily reconnect like Germany) stale states can hang around.

This fix is sort of like that, though needs to run when a gateway is marked down.

Well, after a filter_configure() run we can simply loop through gateways checking for any that are down. if they are down zap the states. That should be pretty easy. I'll take care of it.

Just be aware that the ip specified on pfctl -b is the ip of the interface not of the gateway.

Bascially it is meant like this:
$WAN has ip=192.168.10.10 gw=102.168.10.1
$WAN reloads and gets ip=192.168.10.9 gw=192.168.10.1
the pfctl command in this case should be pfctl -b 192.168.10.10 and not pfctl -b 192.168.10.1.

This is the hard part of it.
That is why i said that i handled this for PPTP/PPPoE/L2TP.
For dhcp it can be added in the script to check if the ip has changed.
For the static ip/carp cases it need to happen on POST of interfaces.php where you can still access the old ip in the config(before being overwritten by the new one).
For openvpn it should be handled by the link down script. But i saw that Seth now call filter_configure() in that case and that needs to be replaced by a new script doing both.

I will study this more thoroughly when i have more time.

What you have checked in is not wrong(though you need to specify the interface ip).
Though i would say that if this is the way to handle this, through the gateway status detection code, it is better to modify apinger or whatever script is run to get this info and teach it to run the pfctl -b command.

If you agree i will see to modify that path/code?!

Huh!? What if an interface use more than that one gateway? It's going to whipe all the states for that interface? This seems wrong to me.

I do not have a environment now but that code seems to grown in home.
The return_gateways_array() return even the interface where the gateway is supposed to be.
No need to go at that length for finding the interface.
Seth can help there!

Yep, you are right. gateway['interface'] had the goods so I adjusted the code. Please test!

with latest snapshot voip phone states doesn't get cleared. Do I need to tick a box somewhere?

Here is what I have found out: the code that runs in delete_states_for_down_gateways() is not working correctly, at the least for pppoe (which is what I have.) It does this:

if($gateway['monitor'] == "down") {

But from debug message I put in, I can see the state of the gateway is 'dynamic', not 'down'. That is not the only problem though - when I tried printing out the IP address as it is fetched for the pfctl command like this:

$int_ip = get_interface_ip($gateway['interface']);

at the point this routine executes, the pppoe interface has (I think) been killed, so there is no IP address to fetch. I was trying to figure out where this all happens, but got kind of lost. It would be nice to get this working, but for the moment, I think I might just patch the code that runs on interface bring-up to do 'pfctl -b' on the LAN IP of the asterisk server, as opposed to the WAN IP of the gateway. It isn't like that address is going to be changing randomly.

I should clarify - seems only the NAT states are still around.

Can you please be more clear what you mean by nat states?

I still have this issue on a August 4 build

udp    87.54.25.131:5060 <- 192.168.11.20:5060    MULTIPLE:MULTIPLE
udp    192.168.11.20:5060 -> 192.168.102.100:5060 -> 87.54.25.131:5060    MULTIPLE:MULTIPLE

I have done some testing regarding this issue around 24th of June.

Calling pfctl -b manually did not do enough at that time. It cleared only 1 of the two states in #26 (the 2nd one, which has gateway address in it: pfctl -b 192.168.102.100).
Only the thing is the kernel seems to instantiate the 2nd rule again when another packet is sent. Only if both rules got removed they would remain gone.

See for more extensive info my post on the forum http://forum.pfsense.org/index.php/topic,26235.msg136537.html#msg136537

Has the behavior of pfctl -b been changed since June 24th? If so I could test it again.

In my last post I ment comment 26, it seems redmine linked issue 26 for the text "poundsign26".

Ok, had some time spare and tested it with today's snapshot (built on Sun Aug 8 10:20:52 EDT 2010) and behaviour is still the same. Only the 2nd state gets killed by pfctl -b and is recreated on the next packet.

So running pfctl -b manually does not seem to work ok either.

I have tested with latest snapshot built on Tue Aug 10 22:06:38 EDT 2010 which includes the change.
It seems failover is broken now. Same for gateway applet on front page and Status->gateway too. Might be related to the apinger output change?

From what I can tell this latest change will make sure pfctl -b is using the correct IP, which of course is also needed for the full fix.
However pfctl -b src_ip fails to clear enough states if NAT is involved.

The following test is done before the last update, since the last update does not seems to have broken the failover.
On a client on the LAN I started a ping to ip 74.125.95.93, which sends a ping packet every second.
This results in the following states in the firewall

all icmp 74.125.95.93:23567 <- 10.10.10.10 0:0
all icmp 10.10.10.10:23567 -> 192.168.103.2:53279 -> 74.125.95.93 0:0

Then I made the gateway on interface 192.168.103.2 go down, watched the route-to rule update in /tmp/rules.debug and I manually ran pfctl -b 192.168.103.2.
For a very short moment only the first of the 2 rules remain in the table, the 2nd one was wiped out by the pftcl -b command.
However on the next ping packet from the client the 2nd rule is automatically added again.

pfctl -b 192.168.103.2 should have wiped out both of the rules.

There is not yet a new enough build to have these latest changes. The next one may possibly have all the changes, but it might not if the builder needs to be restarted to pick up the latest change from the builder tools repository.

Updated to 2.0-BETA4 (i386) built on Sun Aug 15 18:27:24 EDT 2010
Gateway applet working again and the rules.debug file indicates fail over is happening again (gateway which went down is removed from the groups route-to rule).

However, pfctl -b is still not removing enough states.

The following hack works:
grep the statetable before calling pfctl -b with the ip also used for pfctl -b to find all nat states in the form of:
all (proto) (ip1) -> pfctl_-b_IP (:port) -> (ip2)

And then doing a pfctl -k on all ip2 ip1 from all matches, however this will also kill connections from ip1 -> ip2 that might exist over the other gateways that are still up.

Should have been "pfctl -k ip1 -k ip2" in my last comment.

Actually no, pfctl is killing the right states. That is all it has to kill.
Can you please show me the src nodes states after running the pfctl -b command.

Should be 'pfctl -v -s Sources'

I suspect that it might have an impact on this.

This command seems to be giving no output at all, whether I run it before or after the pfctl -b.

Try latest snapshots more state killing added.

Tried the extra state kills on the CLI since snapshot hasn't been updated yet:

Still doesnt do enough, for the ping i'm doing from the LAN ip 10.10.10.144 to 74.125.95.93 I get two states:
icmp 74.125.95.93:63260 <- 10.10.10.144 0:0
icmp 10.10.10.144:63260 -> 192.168.102.2:34763 -> 74.125.95.93 0:0

After running
pfctl -b 192.168.102.2 ; pfctl -K 192.168.102.2 ; pfctl -k 192.168.102.2 ; pfctl -k 0.0.0.0/0 -k 192.168.102.2 ;
killed 2 states from 1 gateway
killed 0 src nodes from 1 sources and 0 destinations
killed 0 states from 1 sources and 0 destinations
killed 0 states from 1 sources and 1 destinations

Only the pfctl -b kills states. One state is from the apinger and the other is
the state below.
icmp 10.10.10.144:63260 -> 192.168.102.2:34763 -> 74.125.95.93 0:0

The other state remains. When the next ping comes in, both states are back and the ping still times out.
Is there any command run yet that kills the following state?
icmp 74.125.95.93:63260 <- 10.10.10.144 0:0

1. pfctl -k 10.10.10.144 -k 74.125.95.93
   killed 2 states from 1 sources and 1 destinations

ping starts working again.

Ok got a deal at it with your help.
We have to kill states of the form 'icmp 74.125.95.93:63260 <- 10.10.10.144 0:0' only if they have cached the 'downed' gateway. I implemented this in pf(4) so test latest snap to see if it actually behaves as expected.

Im happy to report we are close to solving this issue!

Manually executing:
pfctl -b 192.168.102.2 -b 192.168.102.1
after the gw in the previous example went down, kills both states now!

However the code in filter.inc is never called, see function filter_delete_states_for_down_gateways() in filter.inc:

147    foreach ($a_gateways as $gwip => $gateway) {
148        if (stristr($status['status'], "down")) {

$status['status'] is not defined and thus this statement will never be true, changing it to $gateway['status'] fixed this.

Second problem is that pfctl -b is called with the 2nd ip wrong:
In the example above it calls pfctl with the following args:

150                                mwexec("/sbin/pfctl -b {$gateway['srcip']} -b {$gwip}");

Try latest snapshot or commits related to this.
they should get the proper ips to pass to pfctl though i plan to improve this more later on.

Almost works, 2nd arg is set correctly with gw ip now, but local ip is missing now.
Line 163 filter.inc

$cmd = "/sbin/pfctl -b {$gateway['srcip']} ";

Should read

$cmd = "/sbin/pfctl -b {$gwstatus['srcip']} ";

After this change fail over works as expected.

Thank you for the support on this.
If you confirm it works as expected we can close this.

Confirmed fail over is working as expected for me in build 2.0-BETA4 (i386) built on Thu Aug 26 09:00:28 EDT 2010
Tested with 3 statically configured (VLAN) interfaces setup to do NAT for the LAN network and put into a gateway group. I'm unable to test it with pppoe.

When wan fails to wan2 it do clear the states. But when wan connection recover states doesn't get cleared and my voip connection can't receive calls.

I disagree strangely enough :). The feature request took somewhat a turn as it could solve other problems, But for me it seems obviously that the voip fail over part also would contain the recover part.
Never the less I do think a solution can make it in 2.0 If a field was available as based on what was outlined in the description.

- Replace afterfilterchange with apinger so states only gets reset when a fail over happens and not on every filter change.
- When states gets cleared it will only happen to a single IP or a group of IP's and not every entry in state table

shocking. ;) anything further from what's already done isn't critical in the vast majority of cases, has an available work around, and is a lot of work to implement with tons of possibilities for introducing problems. We're going RC soon, not going to do anything further than what's already done for now. Feature request on the remaining is #855, if you have other related ideas please add there.

State Killing on Gateway Failure is great. is there a way to the the opposite, like State Killing on Gateway Success.

I have two WAN, one being a wireless WiMAX connection. We have a dozen or so IP phones, and this feature has helped us redirect traffic to WAN2 (Wireless Uplink) the problem is, the WAN2 is very slow. but its better then nothing. is there a way to Kill the States when WAN comes back up online. that way the voip will route back to the original default route (WAN)