Bug 1781903

I had a look on this and was able to reproduce everything locally, and I think I now understand the root cause of the issue.

 There was a recent rebase (used higher upstream version) of the "nss" and "nss-softokn" version we ship with RHEL-6 to satisfy the requirements for Firefox 68 ESR (BZ#1743628); the packages were released as part of RHEA-2019:3280 [1], . However, while the updated versions brought many improvements in, there appears to be a regression with "libreswan".

Libreswan 3.15 uses a hard-wired "exponent" (e = 3) for the rsasigkeys: 

File: rsasigkey.c
75: #define E       3               /* standard public exponent */
76: /* #define F4	65537 */	/* possible future public exponent, Fermat's 4th number */
... lines omitted ...
416: /*
417:  * generate an RSA signature key
418:  *
419:  * e is fixed at 3, without discussion.  That would not be wise if these
420:  * keys were to be used for encryption, but for signatures there are some
421:  * real speed advantages.
422:  * See also: https://www.imperialviolet.org/2012/03/16/rsae.html
423:  */
424: void rsasigkey(int nbits, int seedbits, char *configdir, char *password)
425: {
426: 	SECStatus rv;
427: 	PK11RSAGenParams rsaparams = { nbits, (long) E };


But the newer nss-softokn-freebl insists on a F4 exponent (65537) as the minimum:

File: rsa.c
265: /*
266: ** Generate and return a new RSA public and private key.
267: **  Both keys are encoded in a single RSAPrivateKey structure.
268: **  "cx" is the random number generator context
269: **  "keySizeInBits" is the size of the key to be generated, in bits.
270: **     512, 1024, etc.
271: **  "publicExponent" when not NULL is a pointer to some data that
272: **     represents the public exponent to use. The data is a byte
273: **     encoded integer, in "big endian" order.
274: */
275: RSAPrivateKey *
276: RSA_NewKey(int keySizeInBits, SECItem *publicExponent)
277: {
... lines omitted ...
290:     /* Require key size to be a multiple of 16 bits. */
291:     if (!publicExponent || keySizeInBits % 16 != 0 ||
292:         BAD_RSA_KEY_SIZE((unsigned int)keySizeInBits / 8, publicExponent->len)) {
293:         PORT_SetError(SEC_ERROR_INVALID_ARGS);
294:         return NULL;
295:     }
296:     /* 1.  Set the public exponent and check if it's uneven and greater than 2.*/
297:     MP_DIGITS(&e) = 0;
298:     CHECK_MPI_OK(mp_init(&e));
299:     SECITEM_TO_MPINT(*publicExponent, &e);
300:     if (mp_iseven(&e) || !(mp_cmp_d(&e, 2) > 0)) {
301:         PORT_SetError(SEC_ERROR_INVALID_ARGS);
302:         goto cleanup;
303:     }
304: #ifndef NSS_FIPS_DISABLED
305:     /* Check that the exponent is not smaller than 65537  */
306:     if (mp_cmp_d(&e, 0x10001) < 0) {
307:         PORT_SetError(SEC_ERROR_INVALID_ARGS);
308:         goto cleanup;
309:     }
310: #endif


Running the "ipsec newhostkey" ("/usr/libexec/ipsec/rsasigkey") under gdb shows:

Breakpoint 1, RSA_NewKey (keySizeInBits=4096, publicExponent=0x7fffffffc6e0) at rsa.c:306
306	    if (mp_cmp_d(&e, 0x10001) < 0) {

(gdb) print e
$1 = {sign = 0, alloc = 64, used = 1, dp = 0x5555557c2d60}
(gdb) print e->dp
$2 = (mp_digit *) 0x5555557c2d60
(gdb) print *e->dp
$3 = 3

(gdb) next
307	        PORT_SetError(SEC_ERROR_INVALID_ARGS);

... a few nexts later:

ipsec rsasigkey: key pair generation failed: "-8187"
567	}
(gdb) 
main (argc=13, argv=0x7fffffffe3b8) at /usr/src/debug/libreswan-3.15/programs/rsasigkey/rsasigkey.c:413
413		exit(0);


Finding the error code explanation:

File: secerr.h
10: #define SEC_ERROR_BASE (-0x2000)
11: #define SEC_ERROR_LIMIT (SEC_ERROR_BASE + 1000)
12: 
13: #define IS_SEC_ERROR(code) \
14:     (((code) >= SEC_ERROR_BASE) && ((code) < SEC_ERROR_LIMIT))
15: 
16: #ifndef NO_SECURITY_ERROR_ENUM
17: typedef enum {
18:     SEC_ERROR_IO = SEC_ERROR_BASE + 0,
19:     SEC_ERROR_LIBRARY_FAILURE = SEC_ERROR_BASE + 1,
20:     SEC_ERROR_BAD_DATA = SEC_ERROR_BASE + 2,
21:     SEC_ERROR_OUTPUT_LEN = SEC_ERROR_BASE + 3,
22:     SEC_ERROR_INPUT_LEN = SEC_ERROR_BASE + 4,
23:     SEC_ERROR_INVALID_ARGS = SEC_ERROR_BASE + 5,


Error is SEC_ERROR_BASE + 5 = -0x2000 + 5 = -8192 + 5 = -8187 = SEC_ERROR_INVALID_ARGS


And while it is nss-softokn causing the regression, I believe it would be easier to fix in "libreswan".

Created attachment 1644002 [details]
Use 65537 as the public exponent.

I wrote this quick patch that uses 65537 as the public exponent. Upstream has used a similar approach, but has also other changes, e.g. bundle() is missing.

I have modified bundle() to support and use 3 octets long public exponents.

Comment on attachment 1644002 [details]
Use 65537 as the public exponent.

This patch or something very similar to this is the right way to fix this issue

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0257