pfSense

Setup:
1. IPSec, IKEv1 site to site tunnel, PSK, Main mode. FQDN identifier - talking to a Mac OS server (racoon)
2. L2TP/IPSec for mobile clients, PSK, Main mode.

Procedure:
1. The IPSec tunnel was set-up, all went well, tunnel established (no other VPN services yet)
2. The L2TP was set-up according to this guide [url]https://doc.pfsense.org/index.php/L2TP/IPsec[/url], however I needed to add "Any TCP flags" to my LAN and L2TP allow-all firewall rule to get connections to work (the floating rule for L2TP was not sufficient). L2TP Clients could connect.

All worked.... until the IPSec tunnel rekeyed a few hours later. Then impossible to reestablish the connection.
Debug logs showing that it was failing at the end of phase 1, hash not matching when I switched to aggressive mode, payload length incorrect in main mode. All DH shared secrets and auth data to hash were identical. This pointed to a PSK problem, yet the connection was established, so I knew my PSKs were correct.

After much head scratching, rebooting of servers, it turns out that the problem is that pfSense/strongSwan was selecting the wrong PSK.

Doing a cat /var/etc/ipsec/ipsec.secrets in the shell gives

 %any : PSK 0s<base 64 encoded L2TP PSK>
 vpn.mycompany.com : PSK 0s<base 64 encoded tunnel PSK>

It would seem that the %any has priority over the identified entry, so the L2TP PSK is always selected - according to the L2TP doc, any user is needed to configure L2TP... which would explain why the tunnel came up to start with (no other PSK candidates).
As a workaround, I changed the tunnel PSK to be the same as the L2TP PSK and, bingo, the tunnel came up.

It would seem that the PSK values are checked in the order they are listed, no hierarchization of matching identifiers, so the %any entry enumerated first, matches and returns the PSK, which is wrong for the IPSec tunnel.

Checking the strongSwan sources (5.5.0)
strongswan-5.5.0/src/libstrongswan/credentials/sets/mem_cred.c:504 -
create_shared_enumerator uses enumerator_create_filter which stops on the first match, only owners within each line in ipsec.secrets have a best match applied via shared_filter (line 454)

To correct:
Either:
1. Within pfSense would be to order secrets in ipsec.secrets so that wildcard ids are at the end,
or
2. Correct strongSwan to choose the best match rather that enumerating the secrets in a linear manner.