Project

General

Profile

Actions

Bug #15449

closed

IPsec VTI static routes may not be added after the system boots

Added by Larry Fahnoe 7 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Routing
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
24.11
Release Notes:
Default
Affected Version:
Affected Architecture:
All

Description

I have a pair of 4200s which were running 23.09.1
Both have an old gateway in a disabled state (see Disabled gateway.png)
There is an IPsec VTI between the 4200s & thus each has a static route to the remote network (Static route.png)

After upgrading to 24.03 I deleted the unused gateway, but upon reboot the static route is not loaded, thus traffic to the remote site doesn't flow. Initially I thought this was due to the state policy change but this is not the case; traffic flows properly as long as the static routes are in place on each end.

This is happening on both systems and is easy for me to replicate: simply roll back to the config with the disabled gateway, reboot (part of the restore) delete the gateway and then reboot. If I leave the disabled gateway in place, the static route is loaded properly upon boot.

The attached screenshots should be self explanatory & the attached system.log covers the boot where the static route was not applied. I have not identified any messages that accompany the problem of the missing route.

--Larry


Files

Disabled gateway.png (100 KB) Disabled gateway.png Shows the old, disabled gateway Larry Fahnoe, 04/29/2024 03:11 PM
Static route.png (47.5 KB) Static route.png Shows the static route Larry Fahnoe, 04/29/2024 03:12 PM
Routes before reboot.png (166 KB) Routes before reboot.png Highlighting the static route before reboot Larry Fahnoe, 04/29/2024 03:15 PM
Routes after reboot.png (158 KB) Routes after reboot.png Showing the route missing Larry Fahnoe, 04/29/2024 03:15 PM
system.log (32 KB) system.log Log of "after reboot" when route went missing Larry Fahnoe, 04/29/2024 03:28 PM
15449.txt (1.45 KB) 15449.txt Marcos M, 05/31/2024 04:03 PM
Actions #1

Updated by Jim Pingle 7 months ago

  • Project changed from pfSense Plus to pfSense
  • Category changed from Configuration Backend to Routing
  • Target version set to 2.8.0
  • Affected Plus Version deleted (24.03)
  • Plus Target Version set to 24.07
Actions #2

Updated by Larry Fahnoe 7 months ago

Additional information.

The gateway that is disabled was originally used with a fiber provider's ONT/router which was operating as a router, thus the WAN int and WANGW gateway had static RFC-1918 addressing. When I reconfigured the ONT/router to operate as a bridge, I switched the WAN int from Static IPv4 to DHCP, which caused pfSense to create a new gateway WAN_DHCP which is what is currently being used. Thinking I might need to go back to using the router I disabled rather than deleted the WANGW.

Now, even though the WANGW is disabled, I'm noting that there is a dpinger trying to monitor an IP that is no longer reachable, so I go to edit WANGW to disable gateway monitoring. System won't allow that because the static IP is no longer within the interface's network, so I blanked out the IP and disabled gateway monitoring.

Upon rebooting, the static route is now gone once again.

Actions #3

Updated by Azamat Khakimyanov 6 months ago

I've tested on 23.09.1
- I've added disabled WAN gateway which is not in the same subnet as a real WAN subnet is
- then I added VTi IPsec tunnel, assigned it and enabled.
- then I added static route via this IPsec VTi
Finally I've updated 23.09.1 up to latest 24.03 and I wasn't able to reproduce this issue with static route not being loaded after deleting disabled WAN gateway.

Please delete wrong gateway, delete this Routed IPsec tunnel and recreate it from scratch.
If you still have issue with static routes via IPsec VTi, please open a ticket on our support platform: https://portal.netgate.com

Actions #4

Updated by Marcos M 6 months ago

  • Status changed from New to Incomplete
Actions #5

Updated by Larry Fahnoe 6 months ago

So on one of the 4200s running 24.03 I have done the following:

1. Deleted static route to 192.168.5.0/24
2. Deleted ALEX_MPLS IPsec VTI interface assignment
3. Deleted IPsec P1 and P2
4. Rebooted

1. Add IPsec P1
2. Add IPsec P2
3. Apply changes
4. Add interface OPT3 ipsec1 (IPsec VTI: Mpls), enable, rename to ALEX_MPLS
5. Apply changes
6. Add static route to 192.168.5.0/24 via ALEX_MPLS
7. Apply changes
8. Routes are present as expected and can ssh to remote pfSense via IPsec
9. Reboot

ALEX_MPLS interface is up and both sides are pingable
Static route to 192.168.5.0/24 does not show up which is the problem I'm attempting to document

# netstat -in | grep ipsec1
ipsec1    1400 <Link#9>          ipsec1                                387     0     0      117     0     0
ipsec1       - fe80::%ipsec1/64  fe80::92ec:77ff:fe8e:66e4%ipsec1        0     -     -        2     -     -
ipsec1       - 192.168.8.0/30    192.168.8.2                           114     -     -      112     -     -
# 
# netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            47.7.216.1         UGS        igc3
47.7.216.0/21      link#4             U          igc3
47.7.222.194       link#6             UHS         lo0
70.59.119.49       47.7.216.1         UGHS       igc3
127.0.0.1          link#6             UH          lo0
192.168.3.0/24     link#3             U          igc2
192.168.3.1        link#6             UHS         lo0
192.168.8.1        link#9             UH       ipsec1
192.168.8.2        link#6             UHS         lo0
192.168.10.1       link#6             UH          lo0
# 
# ping -qc 2 192.168.8.1 
PING 192.168.8.1 (192.168.8.1): 56 data bytes

--- 192.168.8.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 24.621/24.764/24.907/0.143 ms
# 
# ping -qc 2 192.168.8.2
PING 192.168.8.2 (192.168.8.2): 56 data bytes

--- 192.168.8.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.255/0.289/0.322/0.033 ms

This configuration was working with 23.09.1. I will attempt to open a ticket per your request, however I only have TAC-lite.

--Larry

Actions #6

Updated by Lev Prokofev 6 months ago

Ticket for reference #2703470963 the SOs and steps included.

Actions #7

Updated by Lev Prokofev 6 months ago

I'm not able to replicate it on 24.03.

Actions #8

Updated by Lev Prokofev 6 months ago

I finally replicated the issue by restoring the config from the status output file, the root cause is still unknown however I was able to find a way to fix the behavior:

1) Move the static route to another gateway or delete it
2) Delete the IPsec gateway
3)Re-create IPsec gateway
4)Move/set a static route to the IPsec gateway

Actions #9

Updated by Steve Wheeler 6 months ago

  • Status changed from Incomplete to Confirmed
Actions #10

Updated by Azamat Khakimyanov 6 months ago

I used customer's status output file to create the same config on my lab (as Lev done) but I still wasn't able to reproduce this issues.

According to status_output-24.03-after-upgrade.tgz (ticket #2703470963):
customer deleted disabled/wrong gateway
May 8 18:07:32 pfs-a php-fpm408: /system_gateways.php: Configuration Change: (Local Database): Gateways: removed gateway 0

and then customer rebooted pfSense at 18:35
May 8 18:35:08 pfs-a php-cgi482: rc.bootup: Default gateway setting Via Spectrum as default.
May 8 18:35:08 pfs-a php-cgi482: rc.bootup: Gateway, NONE AVAILABLE
May 8 18:35:08 pfs-a php-cgi482: rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1

and one more time at 18:56
May 8 18:56:58 pfs-a php-fpm543: /rc.newwanip: rc.newwanip: on (IP address: 47.7.222.194) (interface: WAN[wan]) (real interface: igc3).
May 8 18:56:58 pfs-a php-cgi630: rc.bootup: dpinger: status socket /var/run/dpinger_WAN_DHCP~47.7.222.194~47.7.216.1.sock not found
May 8 18:56:58 pfs-a php-cgi630: rc.bootup: Default gateway setting Via Spectrum as default.
May 8 18:56:58 pfs-a php-cgi630: rc.bootup: Gateway, NONE AVAILABLE
May 8 18:56:58 pfs-a php-cgi630: rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1

so for me it looks like removing disabled/wrong WAN gateway which shouldn't exist at all, partly ruined pfSense functionality.

I would check if one more test/check should be added which won't allow to add wrong gateways at booting or more precise test before deleting gateways.

Actions #11

Updated by Larry Fahnoe 6 months ago

Another customer is experiencing related issues, see https://forum.netgate.com/topic/188214/vti-gateways-in-24-03 beginning with the posts from May 17 2024

At this point I do not believe this is caused by deleting the disabled gateway. As mentioned in the support ticket dialog, I rolled back to 23.09.1, deleted the disabled gateway WANGW, observed that after rebooting, the static route was loaded and traffic was properly flowing over the IPsec connection. I then upgraded to 24.03 and observed that the static route was NOT being loaded and therefore traffic was not properly flowing over the IPsec connection.

--Larry

Actions #12

Updated by Jim Pingle 6 months ago

  • Plus Target Version changed from 24.07 to 24.08
Actions #13

Updated by Marcos M 6 months ago

  • File 15449.txt 15449.txt added
  • Subject changed from Delete disabled gateway prevents static routes being loaded on boot to IPsec static routes may not added after the system boots
  • Status changed from Confirmed to Ready To Test
  • Assignee set to Marcos M
  • Affected Architecture All added
  • Affected Architecture deleted (amd64)

The inconsistency of the issue seems to stem from the Gateway Monitoring setting. When unchecked (default), the routes will be added as part of the process that happens when a gateway comes online. When checked (monitoring is disabled), no action is taken and hence the routes are not added.

The root of the issue is that when a tunnel is set up, the VTI may not yet be in the interface cache (e.g. after rebooting) and hence adding the route fails (since it's based off the cache). I've attached a patch that should resolve the issue.

Actions #14

Updated by Marcos M 6 months ago

  • Status changed from Ready To Test to Feedback
  • % Done changed from 0 to 100
Actions #15

Updated by Marcos M 6 months ago

  • Status changed from Feedback to Resolved
Actions #16

Updated by Lev Prokofev 6 months ago

The patch is working, confirmed in ticket #2703470963 and on my test device. The issue can be marked as resolved.

Actions #17

Updated by Marcos M 5 months ago

  • Subject changed from IPsec static routes may not added after the system boots to IPsec static routes may not be added after the system boots
Actions #18

Updated by Marcos M 5 months ago

  • Subject changed from IPsec static routes may not be added after the system boots to IPsec VTI static routes may not be added after the system boots
Actions #19

Updated by Jim Pingle about 1 month ago

  • Plus Target Version changed from 24.08 to 24.11
Actions

Also available in: Atom PDF