pfSense

When selecting "Hybrid RSA + xauth" as the authentication mode for an IPsec phase 1 SA on a remote access (mobile) VPN, the following is configured in /var/etc/ipsec/ipsec.conf:

leftauth = xauth-generic
    rightauth = pubkey
    rightauth2 = xauth

The left is the pfsense server and the right is the client.

According to what I've read about how hybrid mode should work (https://tools.ietf.org/html/draft-ietf-ipsec-isakmp-hybrid-auth-05), the server is supposed to authenticate itself with a certificate and the client then authenticates itself to the server using xauth over the encrypted channel that was already established. The above config shows the client authenticating itself with a certificate (rightauth=pubkey) and subsequently authenticating with xauth (rightauth2=xauth). I'm not sure what effect the "leftauth = xauth-generic" has. That method is defined in the local strongswan.conf and calls /etc/inc/ipsec.auth-user.php. It's not clear how the server could authenticate itself to a client by calling that script. I think that, if this works at all, it works as a side effect of also having leftcert=/var/etc/ipsec/ipsec.d/certs/cert-1.crt set in ipsec.conf.

I was unable to connect to a VPN configured with "Hybrid RSA + xauth" using the iOS 9 IPsec client with IKEv1. The client showed an error message saying "The VPN server did not respond". The server logged messages saying:

Sep 25 19:39:10 charon: 11[IKE] <24> found 1 matching config, but none allows HybridInitRSA authentication using Aggressive Mode
Sep 25 19:39:10 charon: 11[IKE] <24> found 1 matching config, but none allows HybridInitRSA authentication using Aggressive Mode
Sep 25 19:39:10 charon: 11[ENC] <24> generating INFORMATIONAL_V1 request 2396490007 [ N(AUTH_FAILED) ]

ipsec.conf has "aggressive=yes" set, but the daemon may ignore this and classify that connection as requiring main mode since it is configured to expect the client to authenticate with a certificate.

I am able to successfully connect an iPhone running iOS 9 using "hybrid RSA + xauth" with IKEv1 by manually altering the ipsec.conf file to have:

leftauth = pubkey
rightauth = xauth

and then running 'ipsec reload'.

In order to get an iPhone to use hybrid authentication to test this, you need to tell it to use shared secret (set AuthenticationMethod to SharedSecret in a configuration profile or don't enable Certificate authentication if configuring with the GUI) and set the Group Name in the GUI (LocalIdentifier in a configuration profile) to whatever the local identifier should be with "[hybrid]" appended to the end. See the following document under "IPSec Dictionary Keys" in the "VPN Payload" section:

https://developer.apple.com/library/ios/featuredarticles/iPhoneConfigurationProfileRef/Introduction/Introduction.html#//apple_ref/doc/uid/TP40010206-CH1-SW27