Paul, is this RFE described somewhere in more detail? I must admin that I am not completely sure what it is about. Is it about DSA peer authentication (ie. authby=rsasig analogy) or something else?

The best documentation is the RFC :)

https://tools.ietf.org/html/rfc7427

   The Internet Key Exchange Version 2 (IKEv2) protocol has limited
   support for the Elliptic Curve Digital Signature Algorithm (ECDSA).
   The current version only includes support for three Elliptic Curve
   groups, and there is a fixed hash algorithm tied to each group.  This
   document generalizes IKEv2 signature support to allow any signature
   method supported by PKIX and also adds signature hash algorithm
   negotiation.  This is a generic mechanism and is not limited to
   ECDSA; it can also be used with other signature algorithms.

Or in other words, if you look at:

https://www.iana.org/assignments/ikev2-parameters/ikev2-parameters.xhtml#ikev2-parameters-12

You will each authentication method having a different registration number. What this RFC does is say, instead of using these specific listed authentication numbers, send algo number 14, and use the SubjectPublicKeyInfo / OIDs to figure out which signature algorithm is really used.

So for RSA, instead of sending algo 1, we will send algo 14 and the proper SBKI / OIDs.

The advantage is that IKE/IPsec does not have to release a new RFC for each new signature algorithm defined. We can just use any (new) PKIX method.

it is expected that this table at IANA will never get any new authentication algorithm ever again. And any new signature algorithms will use algo 14.

This is currently not yet widely supported, but is expected to become the default (as stated as well in rfc4306bis)

OK, thanks a lot. It is definitely clearer now. Is there a way how to check that it works? Can I see algo number in the logs? AFAIK it should be possible now to use an algorithm which is not specified in IKEv2 protocol to see that it works, right? For instance ECDSA different than 9, 10, 11, eg. ECDSA with SHA-224? Or do you have any better tips?

libreswan currently does not implement this, and it also does not implement 9,10 or 11. The idea was to skip implementing 9,10,11 and go straight to digital signatures using algo 14 for those.

Currently, libreswan also does not implement RSA using algo 14 digital signatures either, which would be the first step to implement this RFC.

You can see the algorithm in the logs in the IKE_AUTH parsing/emitting:

| *****emit IKEv2 Authentication Payload:
|    next payload type: ISAKMP_NEXT_v2SA (0x21)
|    flags: none (0x0)
|    auth method: IKEv2_AUTH_RSA (0x1)

FTR, issue was resolved on call with Paul. New libreswan has tighter requirements for certificates - it requires subject alternative name extension. In my previous comment I made a mistake and mention that subjectAltName was set which was incorrect. 

Successfully reproduced and verified on all supported architectures except ppc64le on rhel-alt (not enough HW resources).

Old (libreswan-3.20-5.el7_4)
============================
# ipsec auto --up test-rsasig'
002 "test-rsasig" #1: initiating v2 parent SA
133 "test-rsasig" #1: STATE_PARENT_I1: initiate
002 "test-rsasig" #1: test-rsasig IKE proposals for initial initiator (selecting KE): 1:IKE:ENCR=AES_GCM_C_256;PRF=HMAC_SHA2_512,HMAC_SHA2_256,HMAC_SHA1;INTEG=NONE;DH=MODP2048,MODP3072,MODP4096,MODP8192 2:IKE:ENCR=AES_GCM_C_128;PRF=HMAC_SHA2_512,HMAC_SHA2_256,HMAC_SHA1;INTEG=NONE;DH=MODP2048,MODP3072,MODP4096,MODP8192 3:IKE:ENCR=AES_CBC_256;PRF=HMAC_SHA2_512,HMAC_SHA2_256,HMAC_SHA1;INTEG=HMAC_SHA2_512_256,HMAC_SHA2_256_128,HMAC_SHA1_96;DH=MODP2048,MODP3072,MODP1536 4:IKE:ENCR=AES_CBC_128;PRF=HMAC_SHA2_512,HMAC_SHA2_256,HMAC_SHA1;INTEG=HMAC_SHA2_512_256,HMAC_SHA2_256_128,HMAC_SHA1_96;DH=MODP2048,MODP3072,MODP1536 (default)
133 "test-rsasig" #1: STATE_PARENT_I1: sent v2I1, expected v2R1
002 "test-rsasig" #1: test-rsasig ESP/AH proposals for initiator: 1:ESP:ENCR=AES_GCM_C_256;INTEG=NONE;ESN=DISABLED 2:ESP:ENCR=AES_GCM_C_128;INTEG=NONE;ESN=DISABLED 3:ESP:ENCR=AES_CBC_256;INTEG=HMAC_SHA2_512_256,HMAC_SHA2_256_128;ESN=DISABLED 4:ESP:ENCR=AES_CBC_128;INTEG=HMAC_SHA2_512_256,HMAC_SHA2_256_128;ESN=DISABLED 5:ESP:ENCR=AES_CBC_128;INTEG=HMAC_SHA1_96;ESN=DISABLED (default)
134 "test-rsasig" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
002 "test-rsasig" #2: certificate CN=10.16.66.196,O=Foo,ST=NC,C=US OK
002 "test-rsasig" #2: IKEv2 mode peer ID is ID_IPV4_ADDR: '10.16.66.196'
002 "test-rsasig" #2: negotiated connection [10.16.46.91,10.16.46.91:0-65535 0] -> [10.16.66.196,10.16.66.196:0-65535 0]
004 "test-rsasig" #2: STATE_V2_IPSEC_I: IPsec SA established transport mode {ESP=>0xf87bc307 <0x2949cd16 xfrm=AES_GCM_C_256-NONE NATOA=none NATD=none DPD=passive}

# grep -i 'auth method' /var/log/pluto/pluto.log'
Jan  6 13:57:59: |    auth method: IKEv2_AUTH_RSA (0x1)
Jan  6 13:57:59: |    auth method: IKEv2_AUTH_RSA (0x1)

New (libreswan-3.22-5.el7)
==========================
# ipsec auto --up test-rsasig'
002 "test-rsasig" #1: initiating v2 parent SA
133 "test-rsasig" #1: STATE_PARENT_I1: initiate
002 "test-rsasig" #1: test-rsasig IKE proposals for initial initiator (selecting KE): 1:IKE:ENCR=AES_GCM_C_256;PRF=HMAC_SHA2_512,HMAC_SHA2_256,HMAC_SHA1;INTEG=NONE;DH=MODP2048,MODP3072,MODP4096,MODP8192 2:IKE:ENCR=AES_GCM_C_128;PRF=HMAC_SHA2_512,HMAC_SHA2_256,HMAC_SHA1;INTEG=NONE;DH=MODP2048,MODP3072,MODP4096,MODP8192 3:IKE:ENCR=AES_CBC_256;PRF=HMAC_SHA2_512,HMAC_SHA2_256,HMAC_SHA1;INTEG=HMAC_SHA2_512_256,HMAC_SHA2_256_128,HMAC_SHA1_96;DH=MODP2048,MODP3072,MODP1536 4:IKE:ENCR=AES_CBC_128;PRF=HMAC_SHA2_512,HMAC_SHA2_256,HMAC_SHA1;INTEG=HMAC_SHA2_512_256,HMAC_SHA2_256_128,HMAC_SHA1_96;DH=MODP2048,MODP3072,MODP1536 (default)
133 "test-rsasig" #1: STATE_PARENT_I1: sent v2I1, expected v2R1
002 "test-rsasig" #1: test-rsasig ESP/AH proposals for initiator: 1:ESP:ENCR=AES_GCM_C_256;INTEG=NONE;ESN=DISABLED 2:ESP:ENCR=AES_GCM_C_128;INTEG=NONE;ESN=DISABLED 3:ESP:ENCR=AES_CBC_256;INTEG=HMAC_SHA2_512_256,HMAC_SHA2_256_128;ESN=DISABLED 4:ESP:ENCR=AES_CBC_128;INTEG=HMAC_SHA2_512_256,HMAC_SHA2_256_128;ESN=DISABLED 5:ESP:ENCR=AES_CBC_128;INTEG=HMAC_SHA1_96;ESN=DISABLED (default)
134 "test-rsasig" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
002 "test-rsasig" #2: certificate verified OK: CN=10.16.66.196,O=Foo,ST=NC,C=US
002 "test-rsasig" #2: IKEv2 mode peer ID is ID_IPV4_ADDR: '10.16.66.196'
002 "test-rsasig" #2: negotiated connection [10.16.46.91-10.16.46.91:0-65535 0] -> [10.16.66.196-10.16.66.196:0-65535 0]
004 "test-rsasig" #2: STATE_V2_IPSEC_I: IPsec SA established transport mode 

# grep -i 'auth method' /var/log/pluto/pluto.log'
Jan  6 14:04:20: |    auth method: IKEv2_AUTH_DIGSIG (0xe)
Jan  6 14:04:20: |    auth method: IKEv2_AUTH_DIGSIG (0xe)

For more details see TJ#2237603 and TJ#2237602.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0932