pfSense

I've been really running wireguard through it's paces and I have some thoughts concerning this.

So I have a typical two-node HA cluster and thinking about remote access setups using wireguard. Out of the box, if you point your wireguard remote access clients at your WAN VIP, you won't get traffic. Likely an issue with NAT on egress as the tunnel traffic would appear from the WAN interface IP not the CARP VIP, so the crypto blocks it (I presume). It would be handy to have a few settings though in the WireGuard Tunnel GUI that might make this type of setup easier.

1. Have a checkbox for XMLRPC Sync on a per tunnel basis.
2. Have a drop down for egress NAT address for the wireguard tunnel.

So presumably if you were syncing tunnel configuration between two firewalls, and you also paired it with an automatic NAT rule (based on number 2), for using the WAN CARP VIP for NAT'ing, you could theoretically have a highly available remote access VPN, and the state failover is just handled by who holds the CARP VIP at any given time.

As a general rule, anyone using HA would not be using Automatic Outbound NAT -- they would be using Manual Outbound NAT. So hooking anything automatic into that process isn't viable. Examples of appropriate outbound NAT rules could be added to documentation for using WireGuard with HA.

There are three main scenarios to consider:

• Site-to-Site WireGuard tunnel
  • Would need outbound NAT to force the source of traffic to appear to be the CARP VIP when initiating traffic to a remote peer

• Remote Access WireGuard "server" for multiple remote peers
  • Might need outbound NAT To nudge it to use the correct VIP when responding to clients, but otherwise shouldn't conflict.

• Remote Access WireGuard "client" to a VPN provider or similar
  • Might need outbound NAT to force the source of traffic to appear to be the CARP VIP when initiating but it depends on what the remote end expects.
  • Depending on the use case, for example, XMLRPC could be disabled and each node could have its own keys, remote side could have two peers, etc.

Site-to-Site and "client" roles would cause a conflict as they would still attempt to send traffic out from both nodes and packets would still get to the remote side, but responses would all go to the node whose VIP is set to MASTER. This would at best confuse the remote end and at worst cause traffic to fail as both nodes attempt to send traffic.

I'm not certain how best to solve that yet, but it needs some thought and testing. Could maybe have the CARP up/down script do an ifconfig up/down on the interface so it's configured and present but not used, though I'm not sure how viable or stable that may be.

After testing this for a while in several different configuration styles, it's not viable yet. NAT doesn't help, at best it sort of works but won't fail over. For the time being, it's better off without XMLRPC and with a tunnel built to both nodes. Use dynamic routing of some sort to handle the failover.

See #11354 for more details on what needs to be addressed for this to function.

Until the other issue is addressed, I have noted the limitation here: https://docs.netgate.com/pfsense/en/latest/vpn/wireguard/limitations.html

Jim Pingle wrote in #note-6:

Until the other issue is addressed, I have noted the limitation here: https://docs.netgate.com/pfsense/en/latest/vpn/wireguard/limitations.html

It looks like Feature #11354 was closed a year ago, and released. I am wondering if this feature can have a second look. We use dual 1537 units in HA at 4 of our sites, including production data centers, and I would really like to get away from IPSec, and with the depreciation(rightfully depreciated) of shared secret OpenVPN tunnels, I would prefer WireGuard.

In addition to adding XMLRPC sync, if for anything but to sync the config of remote access clients, I think the solution to the issues above is quite easy, first off, tie the WireGuard service status to the status of a CARP VIP, this will prevent connections from the secondary unit, and the issue with problems with connections from different IPs is easy enough as a firewall rule on any WireGuard tunnels, to only accept inbound connections from the CARP VIP. My understanding then is that if communications are only accepted on the server side CARP VIP, from the client side CARP VIP, if a failover were to happen, there should be no breakdown in communications.

I may be way off base here, but it seems simple enough in my mind, and what was scribbled on paper.

- Marc

EDIT: Reading a bit more, I see what Jim is saying in #note-1, the interface only exists when the service is running(no interfaces are created in haproxy/FRR). What about if there was an option on the Peer Settings(/wg/vpn_wg_peers_edit.php?peer=XXX) that is only available when Dynamic Client is * NOT * checked, that allows the Enable Peer checkbox to be tied to CARP VIP status, this would only need to be on the HA units, and only for S2S Peers.

We are considering switching from OPNsense (because of pfSense better BGP support), which has XMLRPC synchronization enabled for WireGuard and can be used with CARP VIP with a small port forwarding NAT rule workaround, WireGuard kmod and little script (https://gist.github.com/taxilian/eecdc1fb17cf70e8080118cf6d8af412) which disable WG interface for backup node (because BGP advertising). So without this option we are locked.

We have recently switched our site-to-site links to WireGuard, and were disappointed to find that WireGuard settings are not propagated over XMLRPC. I added the settings for WireGuard to the secondary in our HA setup by hand, and fortunately the routes, rules, gateways, and other settings were already there.

Even if there are issues with having the interfaces enabled on the inactive node, propagation of the settings with the requirement to manually (or via script) enable the WireGuard interface(s) would still be a welcome improvement in my opinion as it saves accidentally having these out of sync.