Description of problem:

Libreswan rebase to 4.9 in RHEL-8.8 introduces change (specific only for RHEL-8) that might affect the customers. In RHEL-8.3.0 we introduced bugfix for BZ#1861360. The idea to was keep 1024 bit RSA key working for authby=rsasig. We still the patch for this in RHEL-8.8 [1]. With this patch authby=rsasig is equivalent to authby=rsa-sha1 - PKCS#1 1.5 RSA with SHA1 is used during peer authentication. SHA1 is disabled in RHEL-8, it wasn't a problem until now because until version 4.6 (we had 4.5 in RHEL-8.6.0 and RHEL-8.7.0) libreswan used SHA1 directly through low level NSS API (PK11_SignWithMechanism). But since version 4.7 libreswan uses high level API instead (SGN_Digest) that takes crypto-policy into consideration. And hence SHA1 is rejected (based on the DEFAULT policy). Therefore using authby=rsasig on RHEL-8.8 in DEFAULT policy leads to the following error during peer authentication:

NSS: SGN_Digest(SHA-1) function failed: SEC_ERROR_SIGNATURE_ALGORITHM_DISABLED: Could not create or verify a signature using a signature algorithm that is disabled because it is not secure.

I don't think this is a regression. It only worked before because using SHA1 through the low level API in the previous version bypassed the policy.

I think we should remove this patch and align behavior with upstream and RHEL-9 so that RSA-PSS with SHA2 is used instead of PKCS#1 1.5 RSA with SHA1 for authby=rsasig. If someone needs to keep 1024 bit keys working for some reason it can be done using authby=rsa-sha1. 

[1] https://pkgs.devel.redhat.com/cgit/rpms/libreswan/tree/libreswan-3.32-1861360-nodefault-rsa-pss.patch?h=rhel-8.8.0

Version-Release number of selected component (if applicable):

libreswan-4.9-1.el8

How reproducible:

100%

Steps to Reproduce:

1. Use authby=rsasig peer authencation with RSA keys 

Actual results:

# ipsec auto --up test
181 "test" #1: initiating IKEv2 connection
181 "test" #1: sent IKE_SA_INIT request to 10.16.56.104:500
003 "test" #1: NSS: SGN_Digest(SHA-1) function failed: SEC_ERROR_SIGNATURE_ALGORITHM_DISABLED: Could not create or verify a signature using a signature algorithm that is disabled because it is not secure.
182 "test" #1: sent IKE_AUTH request {cipher=AES_CBC_128 integ=HMAC_SHA2_256_128 prf=HMAC_SHA2_256 group=MODP2048}
003 "test" #1: IKE SA authentication request rejected by peer: AUTHENTICATION_FAILED
036 "test" #1: encountered fatal error in state STATE_V2_PARENT_I2
002 "test" #1: deleting state (STATE_V2_PARENT_I2) aged 0.045762s and NOT sending notification
002 "test" #1: deleting IKE SA but connection is supposed to remain up; schedule EVENT_REVIVE_CONNS

Expected results:

# ipsec auto --up test
181 "test" #1: initiating IKEv2 connection
181 "test" #1: sent IKE_SA_INIT request to 10.0.139.82:500
182 "test" #1: sent IKE_AUTH request {cipher=AES_CBC_128 integ=HMAC_SHA2_512_256 prf=HMAC_SHA2_512 group=MODP2048}
003 "test" #1: initiator established IKE SA; authenticated peer '2048-bit RSASSA-PSS with SHA2_512' digital signature using peer certificate '10.0.139.82' issued by CA 'C=US, ST=NC, O=Foo, CN=CA'
004 "test" #2: initiator established Child SA using #1; IPsec transport [10.0.136.88-10.0.136.88:0-65535 0] -> [10.0.139.82-10.0.139.82:0-65535 0] {ESP/ESN=>0x43eef360 <0x866d012c xfrm=AES_GCM_16_256-NONE DPD=passive}

Additional info:

 * This issues only relevant for RHEL-8, in RHEL-9 we don't have this patch

 * Not fixing this problem will most likely break all configurations using 
   authby=rsasig (in DEFAULT policy)

 * Workaround is simple: 
   - either allow SHA1 using crypto-policy subpolicy
   - or use authby=rsa-sha2 to enforce use of SHA2 instead of SHA1