pfSense

There may be a bug here but the way you're using gateway groups is unnecessary.

Just choose the actual gateway in the firewall rule directly, no need to make a group for that. If only the one gateway is selected, not a group, it can never use any other gateway.

For further reference about the underlying issue though, do you have "Allow default gateway switching" enabled under System > Advanced on the Misc tab?

Hi Jim, thanks for your response, I'll go and check that setting now...

The reason I'm using a gateway group is that I plan on adding a number of additional gateways to this group (each one an additional OpenVPN client/server pair linked to a new gateway in pfSense) and I want to have load balancing and/or failover while always ensuring that the default gateway is never used (I don't want my Hulu Plus account to get flagged as being connected to from a non-US IP address! :) ).

Thanks,
Colin

Jim P wrote:

There may be a bug here but the way you're using gateway groups is unnecessary.

Just choose the actual gateway in the firewall rule directly, no need to make a group for that. If only the one gateway is selected, not a group, it can never use any other gateway.

For further reference about the underlying issue though, do you have "Allow default gateway switching" enabled under System > Advanced on the Misc tab?

It would also help to get a copy of /tmp/rules.debug from when it's running normally, and again when the VPN is down and it's not routing traffic as you expect.

"Allow default gateway switching" was/is UNchecked.

I can't do that now (not at home) but will post /tmp/rules.debug tonight, thanks!

Here are the relevant parts that changed in each one, thanks!

##############################################
##### when all gateways are up
##############################################
# Gateways
GWWAN = " route-to ( vxn0 <REMOVED MY PUBLIC IP GATEWAY HERE> ) " 
GWALIENVPS = " route-to ( ovpnc4 10.9.0.5 ) " 
GWCHICAGOVPS = " route-to ( ovpnc1 10.10.0.5 ) " 
GWUS_Gateways = "  route-to { ( ovpnc4 10.9.0.5 )  }  " 

anchor "userrules/*" 
pass  in  quick  on $LAN  proto tcp  from 192.168.1.0/24  to <negate_networks> flags S/SA keep state  label "NEGATE_ROUTE: Negate policy routing for destination" 
pass  in  quick  on $LAN  $GWUS_Gateways  proto tcp  from 192.168.1.0/24 to any port 25  flags S/SA keep state  label "USER_RULE: SMTP TCP 25 to any via US Gateways" 
pass  in  quick  on $LAN  from   $Colins_Dell_XPS_8300  to <negate_networks> keep state  label "NEGATE_ROUTE: Negate policy routing for destination" 
pass  in  quick  on $LAN  $GWUS_Gateways  from   $Colins_Dell_XPS_8300 to any keep state  label "USER_RULE: Colins_Dell_XPS_8300 to any via US_Gateways" 
pass  in  quick  on $LAN  from   $Apple_TV  to <negate_networks> keep state  label "NEGATE_ROUTE: Negate policy routing for destination" 
pass  in  quick  on $LAN  $GWUS_Gateways  from   $Apple_TV to any keep state  label "USER_RULE: Apple_TV to any via US_Gateways" 
pass  in  quick  on $LAN  from 192.168.1.0/24  to <negate_networks> keep state  label "NEGATE_ROUTE: Negate policy routing for destination" 
pass  in  quick  on $LAN  $GWWAN  from 192.168.1.0/24 to any keep state  label "USER_RULE: LAN to any via WAN" 
block  in  quick  on $IPsec  proto tcp  from any to 192.168.1.0/24  label "USER_RULE: Block IPsec client (TCP) to LAN (UDP/VoIP passes)" 
pass  in  quick  on $IPsec  from any  to <negate_networks> keep state  label "NEGATE_ROUTE: Negate policy routing for destination" 
pass  in  quick  on $IPsec  $GWUS_Gateways  from any to any keep state  label "USER_RULE: IPsec client to any via US_Gateways " 
# WANLANCHICAGOVPSALIENVPSIPsecOpenVPN l2tp array key does not exist for  label "USER_RULE" 
pass  in  quick  on $OpenVPN  from any to 192.168.1.0/24 keep state  label "USER_RULE: OpenVPN client to LAN" 
pass  in  quick  on $OpenVPN  from any  to <negate_networks> keep state  label "NEGATE_ROUTE: Negate policy routing for destination" 
pass  in  quick  on $OpenVPN  $GWUS_Gateways  from any to any keep state  label "USER_RULE: OpenVPN client to any via US_Gateways" 
# WANLANCHICAGOVPSALIENVPSIPsecOpenVPN pptp array key does not exist for  label "USER_RULE" 

##############################################
##### when ALIENVPS (part of US_Gateways) is down, should switch to CHICAGOVPS gateway but switching to WAN instead...I see US_Gateways disappears entirely!
############################################## 
# Gateways
GWWAN = " route-to ( vxn0 <REMOVED MY PUBLIC IP GATEWAY HERE> ) " 
GWALIENVPS = "  " 
GWCHICAGOVPS = " route-to ( ovpnc1 10.10.0.5 ) " 

anchor "userrules/*" 
pass  in  quick  on $LAN  proto tcp  from 192.168.1.0/24 to any port 25  flags S/SA keep state  label "USER_RULE: SMTP TCP 25 to any via US Gateways" 
pass  in  quick  on $LAN  from   $Colins_Dell_XPS_8300 to any keep state  label "USER_RULE: Colins_Dell_XPS_8300 to any via US_Gateways" 
pass  in  quick  on $LAN  from   $Apple_TV to any keep state  label "USER_RULE: Apple_TV to any via US_Gateways" 
pass  in  quick  on $LAN  from 192.168.1.0/24  to <negate_networks> keep state  label "NEGATE_ROUTE: Negate policy routing for destination" 
pass  in  quick  on $LAN  $GWWAN  from 192.168.1.0/24 to any keep state  label "USER_RULE: LAN to any via WAN" 
block  in  quick  on $IPsec  proto tcp  from any to 192.168.1.0/24  label "USER_RULE: Block IPsec client (TCP) to LAN (UDP/VoIP passes)" 
pass  in  quick  on $IPsec  from any to any keep state  label "USER_RULE: IPsec client to any via US_Gateways " 
# WANLANCHICAGOVPSALIENVPSIPsecOpenVPN l2tp array key does not exist for  label "USER_RULE" 
pass  in  quick  on $OpenVPN  from any to 192.168.1.0/24 keep state  label "USER_RULE: OpenVPN client to LAN" 
pass  in  quick  on $OpenVPN  from any to any keep state  label "USER_RULE: OpenVPN client to any via US_Gateways" 
# WANLANCHICAGOVPSALIENVPSIPsecOpenVPN pptp array key does not exist for  label "USER_RULE" 

Jim P wrote:
> It would also help to get a copy of /tmp/rules.debug from when it's running normally, and again when the VPN is down and it's not routing traffic as you expect.

I should have added, all I did was disconnect the OpenVPN client behind the ALIENVPS gateway and then waited for the gateway monitoring to detect it as being down, I assume this replicates the link going down due to other reasons though

Colin Sinclair wrote:

Here are the relevant parts that changed in each one, thanks!

[...]

Colin Sinclair wrote:
Also, I tried setting the gateway in my LAN firewall rules to a standalone gateway (i.e. not to a group) and then put the gateway as down, and it shows the exact same behaviour as for a gateway group, it starts using the default WAN instead of just not routing any traffic at all

I also created a gateway group with three gateways in it (one of them being my default WAN and the other two being OpenVPN based VPN clients) and put one of the VPN clients as tier 1, the other as tier 2 and the default WAN as "never" and when the first tier 1 VPN client is down it goes immediately to the default WAN, bypassing the tier 2 VPN client....also tried putting default WAN to tier 3 behind the other two but once again, same behaviour, tier 1 goes down and it goes again to the WAN bypassing tier 2

I should have added, all I did was disconnect the OpenVPN client behind the ALIENVPS gateway and then waited for the gateway monitoring to detect it as being down, I assume this replicates the link going down due to other reasons though

Colin Sinclair wrote:

Here are the relevant parts that changed in each one, thanks!

[...]

I'm experiencing the same issue on 2.2.5. For policy purposes I require redundant OpenVPN tunnnels, and I also need the system to fail closed rather than fail open.

Still an issue on 2.3 alpha. A somewhat ugly and not terribly secure workaround is to add a rule to outbound NAT disabling NAT on the WAN interface for the specific hosts(s) you want to keep VPN-only. At the very least the UI should correctly reflect that setting WAN to "never" in the gateway group doesn't produce the expected behavior... it shouldn't be an option if it doesn't actually work.