pfSense

I should add: these overlapping SAs don't occur for VTI tunnels to AWS. I consistently get only a single phase2 SA for each AWS tunnel:

# At "A end", which also has 4 VTI tunnels to AWS
/usr/local/sbin/swanctl --list-sas | grep 'con[4567]000'
con7000: #4364, ESTABLISHED, IKEv1, XXXX_i* XXXX_r
  con7000: #307070, reqid 7000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-128/HMAC_SHA1_96/MODP_2048
con5000: #4362, ESTABLISHED, IKEv1, XXXX_i* XXXX_r
  con5000: #307072, reqid 5000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-128/HMAC_SHA1_96/MODP_1024
con4000: #4368, ESTABLISHED, IKEv1, XXXX_i* XXXX_r
  con4000: #307068, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-128/HMAC_SHA1_96/MODP_1024
con6000: #4367, ESTABLISHED, IKEv1, XXXX_i* XXXX_r
  con6000: #307059, reqid 6000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-128/HMAC_SHA1_96/MODP_2048

I do have a different problem with AWS tunnels (#10175) - it's the same HA pair for both these tickets.

I just watched this happen in 2.5.0-DEVELOPMENT (amd64) with a configuration straight out of:
https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/ipsec-routed.html

So, no, #9603 does not solve it.

What now? A shrug "it's StrongSwan" is not an acceptable answer.

Was it 2.5.0 on both ends? If either end is 2.4.x, it still could be that side triggering the problem.

Jim Pingle wrote:

Was it 2.5.0 on both ends? If either end is 2.4.x, it still could be that side triggering the problem.

Yes. (You've reproduced it yourself, so this comment is moot. But I didn't want to just leave the question hanging.)

Looking at this again on 2.5.0, now that it's on strongSwan 5.8.4. I do not see any of my VMs with multiple overlapping child SA entries despite them having ~2 weeks of uptime. (Even some still on 12.0-RELEASE vs some on 12.1-STABLE). Looks like one of the changes in recent strongSwan release may have solved it.

I don't see any on 2.4.5 either (strongSwan 5.8.2) but the 2.4.5 systems I have up at the moment all have low uptimes so I can't be certain they just haven't shown the problem yet.

Contrary to my last note, I am seeing this still, but it still appears to be unpredictable. A system that doesn't show it for long stretches of time will have three the next time I check it. Others have them after only a couple hours. Most only stay in the single digits, some end up super high (~70).

Digging deeper in strongSwan most of the times this has happened in the past have been due to the use of IKEv2 with reauthentication and break-before-make, or a race condition where both sides attempt to initiate at nearly the same time.

Since I have been able to reproduce this on a few different pairs of lab systems, I've changed a couple in the following ways:

• Left both sides on reauthenticate, but enabled make-before-break (On the IPsec Advanced Settings tab) -- Note that this behavior must be supported by both peers. If both peers are pfSense, this should be OK.

- OR -

• Disabled reauthentication and enabled rekey instead (On 2.5.0, leave reauth blank and put a value in rekey. On 2.4.5, check Disable Reauth but leave Disable Rekey unchecked.

The race condition should be solved using advice from earlier on this issue. Set one side to responder only. On the initiator side, set the child SA close action to restart/reconnect. On the responder side, set the Child SA close action to Close/Clear.

After 24 hours all pairs I configured in both of these ways still only had one single child SA. But I'll leave them running and check again after a few more days of uptime/rekeys.

There is a small bug on 2.4.x which prevents responder only from working on VTI, I've pushed a fix for that, but it's too late for 2.4.5-p1 unless circumstances dictate a rebuild. It's a one-line change if anyone wants to make it manually, it will appear here on the issue shortly.

I have a GNS3 lab setup with two pfSense VMs connected via IPSec (IKEv2, VTI). Multi-WAN with failover on one side. Whenever a gateway event/failover occurs there, more child SAs are created. Make-before-break is enabled on both sides.

It looks like after a fresh reboot it is capped/limited at two child SAs, can't push it beyond that. However when i manually disconnect the tunnel and click on connect again, i can easily produce as many Child SAs as i want. Basically everytime i suspend and resume the first WAN link (to trigger automatic failover) the number increases. I would assume that if i wait long enough after a reboot, it also works without disconnecting the tunnel first. In any case, make-before-break does not seem to be a viable reliable workaround on 2.4.x

I can't provoke duplicates on the latest 2.5.0 snapshot, seems to be fixed there...

If it happens on disconnect/reconnect that is more likely the race condition case and not the reauth case. I wouldn't expect make-before-break to help there, but the initiator/responder/SA close action may help.

If it's better on 2.5.0 that could either be from the swanctl conversion, a newer strongSwan, or the initiator/responder setting working there when it doesn't work for VTI on 2.4.x. 2.4.5-p1 will have the newer version of strongSwan, you could apply the patch here on the issue to fix "responder only" for VTI, but if it's related to the swanctl changes then you'll have to run 2.5.0 to get that.

All of my test pairs still only have a single SA this morning (2.4.5 and 2.5.0, multiple causes and changes mentioned here). I'll check them again after the weekend.

So is there a final, required set of baseline versions and recommended configuration which can do into the docs? Or at least here?

Side 1: IKEv2, Rekey configured, Reauth disabled, child SA close action set to restart/reconnect
Side 2: IKEv2, Rekey configured, Reauth disabled, responder only set, child SA close action left at default (clear)

That's the best setup I've hit so far, though I still do see multiple SAs on rare occasions it's nowhere near what it was previously.

Did this. Within 48 hours I have six overlapping phase 2s and am in the #11000's in IPsec IDs.
I'm pretty sure it's time to conclude that VTI does not work and needs to be pulled as an option altogether.

Except that it does work, and thousands of people are using is successfully, and pulling it would cause much more harm at this point.

There is clearly some environmental/situational problem here, not a general problem. It doesn't happen to every setup, even practically identical setups vary in behavior for as-yet-unknown reasons.

Something is triggering this in strongSwan, it would be more productive to focus on what is happening to cause the duplication. (For example, stop strongSwan on both ends, setup remote syslog for ipsec logs, start strongswan and monitor it closely and check the logs when the additional SAs start appearing).

Jim Pingle wrote:

Except that it does work, and thousands of people are using is successfully

Are they? Or are they just not noticing because they're not looking? Eventually, those IDs run out and one side or the other no longer receives packets. All the SADs and SPIs check out. They're just not there on the interface. This broken state can't even be repaired by restarting IPsec.

If you want to see it happen, string together eight sites with VTI links and iBGP. In about thirty days, the core falls over and can't get back up without a reboot. Seriously. I've tried. And I don't have time to throw the kernel on the debugger and figure it out. This has been a huge source of embarrassment for me as an advocate and for the product itself since the switch to strongSwan in 2.4.4.

It's pretty evident that VTIs are not behaving correctly. If you don't want to pull it, I'd recommend marking it experimental.

If you want to discuss it, take it to the forum. As I said, there are many people using it with success. It doesn't affect every installation. Without figuring out why, complaining that it's broken is practically useless. I do have systems on which I can reproduce it, but it doesn't happen until they have been up for weeks, so it's hard to catch and difficult to notice exactly when the problem starts. If you can reproduce it faster, by all means, collect the necessary data. But if you are unwilling to assist in debug why it happens in your environment, then consider your complaint noted and stop posting unhelpful comments.

Running out of IDs when reconnecting should not be a problem on 2.4.5-p1 as it contains strongSwan 5.8.4, which includes the fix for https://wiki.strongswan.org/issues/2315 -- If that still affects your setup, then perhaps the bug hasn't really been fixed in strongSwan, but it's not directly related to this issue.

There are several similar bug reports for duplicate SAs in the strongSwan bug tracker, but last I looked, all of them imply it's a settings issue with IKEv2 and reauth, which the settings I recommended above should address. There may be some other problem in strongSwan, but again, it would be more helpful to figure out the specifics, and ultimately it's going to need to be fixed by strongSwan if it's a problem in strongSwan, which it appears to be at the moment (but we don't know for sure.)

Ticket is marked for Feedback. Feedback is being provided.

I have successfully used your patch and suggested settings on a pair of SG-3100s. There i have two tunnels to AWS that work almost flawlessly. However on another cluster (two Dell R210 II servers) i'm consistently getting duplicates over relatively short periods of time. The patch is applied to all systems, and settings are identical.

Since it's so consistent, maybe we can grab interesting logs from there? Which ones do you need? What log levels would be interesting? This issue is a bummer and made us consider moving away from pfSense, which would be a pity because everything else has been awesome for us over the last few years. But site-to-site IPSec (VTI) tunnels will become more and more of a standard thing for us, they have to be reliable...

This thread may also be related https://forum.netgate.com/topic/153791/vpns-disconnecting-reported-memory-issue

I was observing similar behavior on an SG-3100 to XG-1537 VTI tunnel (both 2.4.5-p1 w/ patch and recommended P1 settings). Something that stood out in the logs was the following repeating cycle of events...

Sep 8 17:43:54 rc.gateway_alarm 8970 >>> Gateway alarm: VTIINTERFACE_VTIV4 (Addr:172.31.0.17 Alarm:0 RTT:46.043ms RTTsd:6.019ms Loss:5%)
Sep 8 17:43:54 check_reload_status updating dyndns VTIINTERFACE_VTIV4
Sep 8 17:43:54 check_reload_status Restarting ipsec tunnels
Sep 8 17:43:54 check_reload_status Restarting OpenVPN tunnels/interfaces
Sep 8 17:43:54 check_reload_status Reloading filter
Sep 8 17:44:10 php-fpm 364 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
Sep 8 17:44:10 check_reload_status Reloading filter

While this certainly seems related, it might be a bit different as it resulted in a fresh SA, but traffic was interrupted for a brief moment and OSPF had to spin back up. No explanation for the completely random monitoring alarms. They occurred from the SG-3100's PoV despite no other connectivity issues. In the end, I had to disable the gateway monitoring action, but it's been stable for over 24 hours and it's appropriately maintaining the SAs now.

I just chalked it up to an ARM quirk as I've never had these issues with our Qotom x86 boxes connected to the same XG-1537 with virtually identical configurations.

Could another difference-maker be NAT-T? As reported above, i'm consistently seeing duplicates on a cluster i'm operating. Interestingly, it only happens on Tunnels to AWS, but not on Tunnels to an EdgeRouter and to another pfSense (both under my control). The difference is that the AWS tunnels show NAT-T while the others don't.

Working Example (only one Child SA):

  con1000: #5151, reqid 1000, INSTALLED, TUNNEL, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 94s ago, rekeying in 2677s, expires in 3507s
    in  ccebc395,  15388 bytes,   424 packets
    out cd9a3565,  35256 bytes,   378 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0

Broken Example:

  con4000: #5132, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 2389s ago, rekeying in 172s, expires in 1211s
    in  cbaef5f6,  25106 bytes,   316 packets
    out 79ef4ff7,  62676 bytes,   435 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0
  con4000: #5133, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 2319s ago, rekeying in 241s, expires in 1281s
    in  c4a7ad40,  20995 bytes,   285 packets
    out 3879fb50,  54144 bytes,   376 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0
  con4000: #5134, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 2257s ago, rekeying in 317s, expires in 1343s
    in  cac3b309,   4844 bytes,    86 packets
    out f1fb3f7a,  18704 bytes,   132 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0
  con4000: #5135, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 2234s ago, rekeying in 466s, expires in 1366s
    in  ce391b6d,  14330 bytes,   183 packets
    out c1248764,  36568 bytes,   250 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0
  con4000: #5136, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 2195s ago, rekeying in 362s, expires in 1405s
    in  c9f21daf, 560345 bytes,  7857 packets
    out bf275006, 1507208 bytes, 10486 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0
  con4000: #5146, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 462s ago, rekeying in 2193s, expires in 3138s
    in  c08737f6,  54508 bytes,   710 packets
    out 70d7adba, 129676 bytes,   901 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0
  con4000: #5147, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 315s ago, rekeying in 2222s, expires in 3285s
    in  ca7144e0,  25618 bytes,   401 packets
    out be33b986,  78124 bytes,   545 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0
  con4000: #5148, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 225s ago, rekeying in 2365s, expires in 3375s
    in  c4e86d84,  37407 bytes,   513 packets
    out ead0f31a, 100964 bytes,   703 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0
  con4000: #5150, reqid 4000, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA2_256_128/MODP_4096
    installed 108s ago, rekeying in 2462s, expires in 3492s
    in  c5289559,  36070 bytes,   486 packets
    out c0bab313,  93256 bytes,   650 packets
    local  0.0.0.0/0|/0
    remote 0.0.0.0/0|/0

Then again, i had Tunnels to AWS with identical configuration working on my SG-3100s. But that setup has since been removed so i can't compare the output unfortunately...

I don't think NAT-T is the issue. All my firewalls have public IPs, and my tunnels don't have NAT-T (see status output in original post).

I'm having the same issue with duplicating VTI Phase2s with tunnels to AWS.

Did anyone find settings that fixed this with tunnels to AWS?

This is the response from Amazon. Since they weren't sure about 'Make before break' I will try the other settings they suggest:
________________________________________________________________________________________________________________________

Upon investigation, I also confirmed the multiple duplicated Phase2s of your VPN and randomly re-keying at unexpected time.
In regard to the proposed configuration changes, I assume Side 1 is AWS whereas Side 2 is your CGW device. Please see below comments:

Side 1
Disable Reauthentication
Set to 'Responder Only'

"Reauthentication" is NOT supported by AWS VPN so you don't need to worry about it. It's always going to Rekey instead of Reauthentication.
AWS VPN is in 'Responder Only' by default. However, I see you have configured "Startup Action": Start. Please make sure both "Startup action" and "DPD timeout action" are configured back to below¹[2]:

DPD Timeout Action: Clear
Startup Action: Add

You can make changes to both via VPC Console > Site-to-Site VPN Connections -> (Select your VPN)>Actions->Modify VPN Tunnel Options.

Side 2
Disable Reauthentication
Set 'Child SA close action' to 'Restart/Reconnect'

"Reauthentication" is not supported by AWS VPN, so disabling it on the CGW won't cause any issue.
According to Netgate documentation³, I understand 'Child SA close action' controls how the IPsec daemon behaves when a child SA (P2) is unexpectedly closed by the peer.
If "Restart/Reconnect" is configured on your side, it should eliminate the issue as DPD Timeout Action: Clear and Startup Action: Add are configured at AWS side.

Resource:
[1] Site-to-Site VPN tunnel initiation options - https://docs.aws.amazon.com/vpn/latest/s2svpn/initiate-vpn-tunnels.html
[2] Tunnel options for your Site-to-Site VPN connection - https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNTunnels.html
[3]https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/configure.html#advanced-options

Lastly, Make-before-break option is not available for configuration on AWS side. But I think that is the default behavior.
I hope the above information has addressed all your concerns. Please let me know if the issue persists after you make the above changes. Happy to further assist you on this issue.

Above changes made no difference, Phase2's to AWS just duplicated again, I currently have 187 duplicated Phase2's.

Regarding the issue when the other side is Meraki:
This issue is resolved by changing the Phase1 settings to use "Restart/Reconnect" for the Child SA Close Action.

Jim Pingle What a great find! Seems so logical after reading, but it's one of those why didn't I think of that things. Reminds of https://redmine.pfsense.org/issues/10811 (where Netgate's AutoConfigBackup server was getting hammered due to everyone's config backups triggering at the same time). The proposed fix (not sure if it was ever merged...) was adding a randomized 5-10 minute delay to whatever the base time was.

Glad I read your commit message https://github.com/pfsense/pfsense/commit/170c0f086a6a16aea7688ef841ac1623773ac2e8 ... I was about to suggest "why not do the same for these SA lifetimes?" and of course, you've already done it! A belated Xmas present to us all!

luckman212 wrote:

"why not do the same for these SA lifetimes?"

I'm still considering exposing those options (Rekey Time, Rand Time, and Life Time) in the GUI so they can be fine-tuned for P2 like we have for P1, but yeah they are now set automatically on 2.5.0. On 2.4.5 the old ipsec.conf values let strongSwan do that automatically, on 2.5.0 the new swanctl format needs to be set explicitly. I'm not sure enough people care about setting them to non-default values to make it worthwhile. The difference with P1 is that the user also has to decide between rekey and reauthenticate (or neither) so it's not quite so simple as calculating the values like we can do for P2. If I do that, I'd make a separate issue for it as a feature request.

I'm not sure what, if anything, strongSwan could do here since it can be solved with settings, but I passed along the information I gathered to them as well: https://wiki.strongswan.org/issues/3663

Jim Pingle wrote:

work around it

Well, at least this is the appropriate term. Metaphorically speaking: it's trying to space cars going though an intersection to avoid collision. Unfortunately, an incremental offset will always collide at some point. One just hopes it's far enough in the future not to matter. You can assure the drivers of those cars of that decision.

Hope should not be the basis of a mature technology.

And denial for nearly a year should not be the modus operandi of a reputable manufacturer.

"Well what should we have done?"
I dunno. There's decades of computer science around locking, two-party negotiation, and random backoff mechanisms. Maybe one of those.
"But it's StrongSWAN's fault!"
And who made the decision to abandon KAME-derived systems in favor of porting a known unstable Linux toy? And to release a poorly tested feature?
"This is an open source project, we don't have the resources for maintaining something like that!"
No, you were an open source project. You're making money off this now. You're "professionals." Act like it.

NB: A part of professional behavior is humble self-reflection on followup criticism like this. But reviewing earlier lines of this ticket, I'm sure the response will be either arguing back or summary deletion of the note. That's fine. You may detect some extra bitterness here. It would be because this bug was material to my leaving a rather comfortable job and that former employer's move to PANs. These devices now jeopardize my remaining financial interest in that company. Thanks, Netgate. I've learned a lot.