pfSense

Adding comment per request of NetGate TAC support after working with them via ticket "43897451036". Since this report is closed I also shared the same information in this report: https://redmine.pfsense.org/issues/4591#note-5

It would be nice to be able to specify an additional WAN IP in a tunnel on the pfSense side so that if the primary WAN IP becomes inaccessible pfSense will attempt to establish an IPsec tunnel over the defined secondary WAN IP address.

In this particular scenario, for context:

• There is a peer firewall with two ISPs.
• We have a single pfSense with a single WAN IP address.
• DDNS, with Palo Alto, is too slow to updates, its minimal update window is 1 hour.
• pfSense is currently in responder only mode with policy based tunnels on both sides but leveraging WAN monitoring on the Palo Alto side to send traffic across a different WAN interface if the primary WAN goes down, which works.
  • It would be nice to be able to define a backup WAN IP in a single pfSense tunnel to an endpoint (firewall) with two WAN IPs.*

The currently workaround I arrived that, in this particular scenario between a Palo Alto with two WAN IPs and a pfSense with a Single WAN IP is:

• Retain two tunnels on pfSense.
• On the pfSense side: * Change the P1 on both WAN1 and WAN2 tunnels on pfSense and the following settings under "Advanced Options". * Change "Child SA Start Action" from "Default" to "None (Responder Only)." * Change "Child SA Close Action" from "Default" to "Close connection and clear SA".
• On the Palo Alto side: * Confirm primary tunnel route metric is 10 and "Admin Distance" is "default". * Change secondary tunnel route metric to 100 and "Admin Distance" is default." * In "Network -> Network Profiles -> Monitor" ensure "FailoverWANProfile" is created with an "Action" of "Fail Over" and an interval of "3" (seconds) and a Threshold of "5". * Edit the Primary IPsec tunnel so ONLY IT HAS THE FAILOVER MONITOR! The secondary IPsec tunnel does NOT HAVE/GET THE FAILOVER MONITOR!
• I THINK what was tripping up the tunnel monitoring portion and failing back to WAN 1 is that since both tunnels are in a single VRF (Virtual Routing and Forwarding) interface, it will actively try to establish VPNs over both ISPs.
• If both tunnels are configured with a failover technically they'd both fail to each other if the other is down, now logically, this makes sense, if WAN 2 goes down fail to WAN 1, but we already have that configured in the WAN 1 tunnel monitor.
• So, WAN 1 goes down, tunnel monitor triggers, route gets updated that says use the tunnel on WAN 2 (Palo Alto), then traffic flows over WAN 2 to pfSense. (Keep in mind, pfSense still has two tunnels up to each WAN interface - so two tunnels per site in pfSense) BUT it's the Palo Alto making the route/tunnel priority decision based on the tunnel failover and route metric. (Now, this does go against policy based routing because pfSense isn't aware of what it should send traffic over however the Palo Alto says "well, my WAN 1 tunnel has a metric of 10, so this route will win which means I'm sending IPsec traffic over my WAN1 tunnel." Now, I am presuming, pfSense, on its side, will see traffic coming in that tunnel and is smart enough to send it out the same tunnel even though it has two tunnels with different WANs but with the same P2s." Like, there is some smart routing that takes place.
• What I did NOT do that pfSense recommended yesterday to prevent duplicates of IPsec tunnels but because Palo Alto support has been absolutely miserable to work with it, is, I did NOT change the "Remote Gateway" value to "0.0.0.0" and keep only a single tunnel since, due the way Palo Alto is configured and without having to re-setup/change the VRF (this will break communication everywhere on the Palo Alto), I left two tunnels to each WAN IP on the Palo Alto present on the pfSense so pfSense has two tunnels per location, one per WAN.