Description of problem:
FreeIPA / IdM would like to move from named-pkcs11 to plain Bind9 named with OpenSSL engine. This method uses openssl-pkcs11 / libp11 to interface between an unmodified named binary and SoftHSM2 with DNSSEC keys. An experimental patch works fine on standard system but fails on FIPS enabled systems. named fails to start with error message "/etc/named.root.key:12: configuring managed key for '.': out of memory".

After some debugging I figured out that the problem originates in bind's dst_key_fromdns(), libdns' opensslrsa_fromdn(), and eventually RSA_new(). The latter fails because libp11's RSA method does not have the flag RSA_FLAG_FIPS_METHOD set.

The bug can be easily reproduced on a FIPS system with OpenSSL command line tools.

Version-Release number of selected component (if applicable):
openssl-1.1.1d-2.fc31.x86_64
openssl-pkcs11-0.4.10-5.fc31.x86_64
softhsm-2.5.0-4.fc31.1.x86_64

How reproducible:
always

Steps to Reproduce:
1. put a system into FIPS mode (1minutetip --flavor ci.m1.medium.rng --fips Fedora-31)
2. Create /tmp/openssl.cnf with
[openssl_init]
engines=engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/lib64/pkcs11/libsofthsm2.so
MODULE_PATH = libsofthsm2.so
init = 0

3. OPENSSL_CONF=/tmp/openssl.cnf openssl genrsa -engine pkcs11


Actual results:
engine "pkcs11" set.
Generating RSA private key, 2048 bit long modulus (2 primes)
139740023297856:error:0406A0C8:rsa routines:RSA_new_method:non FIPS rsa method:crypto/rsa/rsa_lib.c:94:

Expected results:
no error

Additional info:

Traceback from Bind9 debug session:

(gdb) n
93          if (FIPS_mode() && !(ret->meth->flags & RSA_FLAG_FIPS_METHOD)) {
(gdb) n
94              RSAerr(RSA_F_RSA_NEW_METHOD, RSA_R_NON_FIPS_RSA_METHOD);

(gdb) bt
#0  RSA_new_method (engine=engine@entry=0x0) at crypto/rsa/rsa_lib.c:94
#1  0x00007ffff7b5575b in RSA_new () at crypto/rsa/rsa_lib.c:22
#2  0x00007ffff7f1be2f in opensslrsa_fromdns (key=0x7ffff4d462f0, data=0x7ffff71c7a80) at ../../../lib/dns/opensslrsa_link.c:1254
#3  0x00007ffff7f20042 in frombuffer (name=name@entry=0x7ffff71c7b00, alg=alg@entry=8, flags=flags@entry=257, protocol=protocol@entry=3, 
    rdclass=rdclass@entry=1, source=source@entry=0x7ffff71c7a80, mctx=0x5555556515a0, keyp=0x7ffff71c7968) at ../../../lib/dns/dst_api.c:1943
#4  0x00007ffff7f20240 in dst_key_fromdns (name=name@entry=0x7ffff71c7b00, rdclass=<optimized out>, source=source@entry=0x7ffff71c7a80, 
    mctx=mctx@entry=0x5555556515a0, keyp=keyp@entry=0x7ffff71c7a08) at ../../../lib/dns/dst_api.c:794
#5  0x00005555555ae41a in dstkey_fromconfig (vconfig=vconfig@entry=0x0, key=0x7ffff72083f8, managed=managed@entry=true, 
    target=target@entry=0x7ffff71c9d88, mctx=mctx@entry=0x5555556515a0) at ../../../bin/named/server.c:792
#6  0x00005555555ae592 in load_view_keys (keys=<optimized out>, vconfig=vconfig@entry=0x0, view=view@entry=0x7ffff00d2cf0, 
    managed=managed@entry=true, keyname=keyname@entry=0x0, mctx=mctx@entry=0x5555556515a0) at ../../../bin/named/server.c:845
#7  0x00005555555af09b in configure_view_dnsseckeys (view=view@entry=0x7ffff00d2cf0, vconfig=vconfig@entry=0x0, config=config@entry=0x7ffff72018a8, 
    bindkeys=bindkeys@entry=0x0, auto_root=auto_root@entry=false, mctx=mctx@entry=0x5555556515a0) at ../../../bin/named/server.c:1041
#8  0x00005555555bbdf6 in configure_view (view=<optimized out>, viewlist=viewlist@entry=0x7ffff71cc8c0, config=<optimized out>, 
    vconfig=vconfig@entry=0x0, cachelist=cachelist@entry=0x7ffff71cc8e0, bindkeys=<optimized out>, mctx=<optimized out>, actx=<optimized out>, 
    need_hints=<optimized out>) at ../../../bin/named/server.c:4705
#9  0x00005555555ca9ff in load_configuration (filename=<optimized out>, server=server@entry=0x7ffff71da010, first_time=first_time@entry=true)
    at ../../../bin/named/server.c:8125
#10 0x00005555555cbb02 in run_server (task=<optimized out>, event=<optimized out>) at ../../../bin/named/server.c:8887
#11 0x00007ffff7cbfebe in dispatch (manager=0x7ffff71d3010) at ../../../lib/isc/task.c:1145
#12 run (uap=0x7ffff71d3010) at ../../../lib/isc/task.c:1319
#13 0x00007ffff79444e2 in start_thread () from /lib64/libpthread.so.0
#14 0x00007ffff76e96a3 in clone () from /lib64/libc.so.6

(gdb) p *ret
$5 = {pad = 0, version = 0, meth = 0x555555631b10, engine = 0x5555556375d0, n = 0x0, e = 0x0, d = 0x0, p = 0x0, q = 0x0, dmp1 = 0x0, dmq1 = 0x0, 
  iqmp = 0x0, prime_infos = 0x0, pss = 0x0, ex_data = {sk = 0x0}, references = 1, flags = 0, _method_mod_n = 0x0, _method_mod_p = 0x0, 
  _method_mod_q = 0x0, bignum_data = 0x0, blinding = 0x0, mt_blinding = 0x0, lock = 0x7ffff0000b60}

(gdb) p *ret->engine
$6 = {id = 0x7ffff5199000 "pkcs11", name = 0x7ffff5199016 "pkcs11 engine", rsa_meth = 0x555555631b10, dsa_meth = 0x0, dh_meth = 0x0, 
  ec_meth = 0x555555631a90, rand_meth = 0x0, ciphers = 0x0, digests = 0x0, pkey_meths = 0x7ffff5196d80 <PKCS11_pkey_meths>, pkey_asn1_meths = 0x0, 
  destroy = 0x7ffff518e760 <engine_destroy>, init = 0x7ffff518e730 <engine_init>, finish = 0x7ffff518e700 <engine_finish>, 
  ctrl = 0x7ffff518e6b0 <engine_ctrl>, load_privkey = 0x7ffff518e600 <load_privkey>, load_pubkey = 0x7ffff518e670 <load_pubkey>, 
  load_ssl_client_cert = 0x0, cmd_defns = 0x7ffff519e720, flags = 0, struct_ref = 7, funct_ref = 5, ex_data = {sk = 0x5555556311e0}, 
  prev = 0x5555556376f0, next = 0x0