Project

General

Profile

Actions

Bug #12808

closed

Wireguard Gateways disabled when Wireguard Service is Manually Restarted

Added by RED SKULL 4 months ago. Updated 18 days ago.

Status:
Resolved
Priority:
Normal
Category:
WireGuard
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Affected Version:
Affected Plus Version:
Affected Architecture:
amd64

Description

If the wireguard service is manually restarted at any time after boot, Wireguard gateways are automatically disabled (also grayed out in the UI) and do not come back up once the tunnels are rebuilt and WG service is restarted. This issue occurs on PfSense 2.6.

/system_gateways.php: GATEWAYS: Group ####### did not have any gateways up on tier #!
/rc.dyndns.update: GATEWAYS: Group ####### did not have any gateways up on tier #!

Manually re-enabling gateways under Routing --> Gateways is needed to establish web traffic on aforementioned WG tunnels.


Files

Screenshot 2022-03-14 at 22.55.10.png (281 KB) Screenshot 2022-03-14 at 22.55.10.png B P, 03/14/2022 08:53 AM
tun_wg0.png (134 KB) tun_wg0.png WireGuard Interface Waqas Khan, 03/22/2022 06:20 AM
WG0GW.png (94.7 KB) WG0GW.png Gateway for WireGuard Waqas Khan, 03/22/2022 06:20 AM
Static.png (37.4 KB) Static.png Static Route Waqas Khan, 03/22/2022 06:20 AM
Actions #1

Updated by RED SKULL 4 months ago

This issue specifically occurs on PfSense 2.6 CE final release.

Once gateways are manually re-enabled, you can see the following in logs(Gateway groups consisting of WG gateways do exist in this configuration):

php-fpm 46388 /system_gateways.php: Configuration Change: (Local Database): Gateways: enable/disable
php-fpm 38242 /system_gateways.php: GATEWAYS: Group ###### did not have any gateways up on tier 1!
php-fpm 38242 /system_gateways.php: GATEWAYS: Group ###### did not have any gateways up on tier 1!
check_reload_status 407 Syncing firewall

then static route for monitor IP is rerouted over the WG tunnel and traffic is then allowed to pass over said tunnel:

/system_gateways.php: Removing static route for monitor X.X.X.X and adding a new route through X.X.X.X

Actions #2

Updated by Jim Pingle 4 months ago

  • Project changed from pfSense to pfSense Packages
  • Category changed from WireGuard to WireGuard
  • Assignee set to Christian McDonald
  • Release Notes deleted (Default)
Actions #3

Updated by Viktor Gurov 4 months ago

Unable to reproduce -
wireguard gateways works as expected after:
1) Restarting the Wireguard service on the Status / Services page
2) Restarting the Wireguard service on the VPN / Wireguard page (restart icon)

Actions #4

Updated by Viktor Gurov 4 months ago

  • Status changed from New to Feedback
Actions #5

Updated by RED SKULL 4 months ago

was testing done with multiple WG gateway groups like in aforementioned setup? Just FYI, WG tunnels had monitor IPs that were not in same subnet as interface e.g. 1.1.1.1.

Actions #6

Updated by Christian McDonald 4 months ago

I'm not able to reproduce this either. Can you post some redacted screenshots of your exact configuration?

Actions #7

Updated by B P 3 months ago

I have the same issue. One side of the Wireguard VPN is disabled after reboot. Both sides of the VPN appear to have the same config. It isn't always disabled, but around 90% of the time it is. I've tried multiple settings, including: Disable Gateway Monitoring, multiple Monitor IP addresses - internal and external. I'm starting to think it's a bug. My device is a Netgate 3100 - 22.01-RELEASE (arm). I'm happy to supply logs / screen captures. Thanks.

Actions #8

Updated by Zep Man 3 months ago

Also see:
https://old.reddit.com/r/PFSENSE/comments/tc8zsx/wireguard_service_not_starting_on_system/

Can also confirm this from my side. Notice that I do not use gateway groups.

Upgraded installation from 2.5.2 to 2.6.0, but it seems to also occur on fresh installations. Gateways assigned to WireGuard interfaces are disabled upon boot most* of the time. It seems to work sometimes (1 out of 10?), as noted by B P in the previous reaction.

I also disabled Gateway Monitoring, Gateway Monitoring Action and enabled the use of a non-local gateway through interface specific route. This all did not make a difference.

My theory for the 90% scenarios:
-Gateways are up.
-System reboots.
-Gateway configuration starts first, notices that WireGuard interfaces are not available.
-Gateway configuration disables gateways that rely on WireGuard interfaces.
-WireGuard configuration starts, sees that gateways are disabled and fails.

My theory for the 10% scenarios:
-Gateways are up.
-System reboots.
-WireGuard configuration starts, sees that gateways are up.
-Gateway configuration starts, sees that WireGuard interfaces are up.

It cannot always be guaranteed that the interface is available when gateway configuration occurs.

Why does pfSense automatically disable gateways? Seems like a recipe for disaster. Have it generate a notice or error if needed. Or alternatively, add an option for a gateway to disable automatic management (or interface checks).

This worked fine in pfSense 2.5.2. I am considering a downgrade or a switch to opnSense.

Actions #9

Updated by Waqas Khan 3 months ago

I am the original author of this post https://old.reddit.com/r/PFSENSE/comments/tc8zsx/wireguard_service_not_starting_on_system/

After banging my head all over the web I finally decided to post the bug here today but was very releived and joyed to see that others have taken up the issue already.

What I believe is happening is when the firewall halts or reboots, it stops the WireGuard service, and in the process the Gateway associated with it gets disabled, the same way if you uninstalled the WireGuard package. However, upon booting back up it doesn't gets enabled automatically and hence the WireGuard service also fails to start as the Gateway associated is down. Once the Gateway is manually enabled, the Service Watchdog brings up the WireGuard service back up in no time. This is my take on this issue.

Maybe this from System Logs may help:

Mar 22 09:22:30 php 422 rc.bootup: Gateway, switch to:
Mar 22 09:22:30 kernel .done.
Mar 22 09:22:30 php 422 rc.bootup: Static Routes: Gateway IP could not be found for 192.168.5.0/24
Mar 22 09:22:30 kernel done.

It could also be just as Zep Man has theorized. Or a combination of the two.

Actions #10

Updated by Waqas Khan 3 months ago

Here are some screenshots for reference.

Note: Disabling Gateway Monitoring and Using Non-local Gateway or using a /28 Subnet in WG0 Interface all lead to same result.

Actions #11

Updated by Scott Lykens 3 months ago

I found this bug after having WireGuard stop passing traffic after a WAN GW went down and came back up. Upon restoration, WireGuard would not speak to the peer nor would it even update DNS for the peer when I tried to use a different endpoint (ipv4 vs ipv6).

After rebooting, I expected everything to come up fine but it did not exhibiting the symptoms described by above commenters - WG looked good and passed traffic between peers but ultimately I found no routes in the kernel routing table for the networks beyond the peers.

In my case, I have two peers, one with network 10.11.0.0/16 and one with network 10.13.0.0/16 behind them, respectively. Here are the relevant boot logs containing gateway information about either network:

Mar 28 23:56:18 gateway php[416]: rc.bootup: Static Routes: Gateway IP could not be found for 10.13.0.0/16
Mar 28 23:56:18 gateway php[416]: rc.bootup: Static Routes: Gateway IP could not be found for 10.11.0.0/16
Mar 28 23:56:34 gateway php[10433]: /usr/local/pkg/wireguard/includes/wg_service.inc: Static Routes: Gateway IP could not be found for 10.13.0.0/16
Mar 28 23:56:34 gateway php[10433]: /usr/local/pkg/wireguard/includes/wg_service.inc: Static Routes: Gateway IP could not be found for 10.11.0.0/16

After boot completed and I could access the GUI, I saved one of the WireGuard routes in the GUI and applied changes - the routes were immediately added and the network behaved as expected.

Actions #12

Updated by Val Mor 19 days ago

I have similar issue on pfSense 2.6.0-RELEASE.
Configured WireGuard tunnel and set a static route.
After reboot of pfSense or restart of Wireguard, static route is gone.
If I go to the System -> Routing -> Static Routes and save same route (without editing anything), the route is added again and everything is working.

Actions #13

Updated by → luckman212 19 days ago

Val Mor if you add the System Patches package and then add a patch using this url:

https://github.com/theonemcdonald/pfSense-pkg-WireGuard/commit/0d40bbc333fb2a769025230ae5eefaf5147b1bac.patch

set Path Strip Count = 4

try to apply that and then re-test, see if it fixes your issue

Actions #14

Updated by Val Mor 19 days ago

→ luckman212 wrote in #note-13:

Val Mor if you add the System Patches package and then add a patch using this url:

https://github.com/theonemcdonald/pfSense-pkg-WireGuard/commit/0d40bbc333fb2a769025230ae5eefaf5147b1bac.patch

set Path Strip Count = 4

try to apply that and then re-test, see if it fixes your issue

Applied the Patch.
Static route survives reboot of pfSense or restart of WireGuard service.
Thanks!

Actions #15

Updated by Christian McDonald 19 days ago

Cherry picked this commit to RELENG_2_6_0 ports tree. Look for a package update.

Edit: v0.1.6_2 is available in CE 2.6.0.

Actions #16

Updated by Christian McDonald 18 days ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF