Bug 1827535 - PKCS11 engine does not work in FIPS mode: RSA_new_method:non FIPS rsa method
Summary: PKCS11 engine does not work in FIPS mode: RSA_new_method:non FIPS rsa method
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openssl-pkcs11
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Anderson Sasaki
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1827602
TreeView+ depends on / blocked
 
Reported: 2020-04-24 06:40 UTC by Christian Heimes
Modified: 2020-05-19 03:48 UTC (History)
2 users (show)

Fixed In Version: openssl-pkcs11-0.4.10-6 openssl-pkcs11-0.4.10-6.fc32 openssl-pkcs11-0.4.10-6.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-01 00:36:23 UTC
Type: Bug


Attachments (Terms of Use)
Experimental FIPS hack (569 bytes, patch)
2020-04-24 08:01 UTC, Christian Heimes
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Fedora Pagure freeipa issue 8094 0 None None None 2020-04-24 06:40:31 UTC
Github freeipa freeipa pull 3774 0 None closed [DNSSEC] Allow using of a custom OpenSSL engine for BIND 2021-02-08 10:18:31 UTC
Red Hat Bugzilla 1745082 1 None None None 2021-01-20 06:05:38 UTC
Red Hat Bugzilla 1777892 0 unspecified CLOSED openssl-pkcs11 does not support using RSA keys in FIPS mode 2021-02-22 00:41:40 UTC

Description Christian Heimes 2020-04-24 06:40:32 UTC
Description of problem:
FreeIPA / IdM would like to move from named-pkcs11 to plain Bind9 named with OpenSSL engine. This method uses openssl-pkcs11 / libp11 to interface between an unmodified named binary and SoftHSM2 with DNSSEC keys. An experimental patch works fine on standard system but fails on FIPS enabled systems. named fails to start with error message "/etc/named.root.key:12: configuring managed key for '.': out of memory".

After some debugging I figured out that the problem originates in bind's dst_key_fromdns(), libdns' opensslrsa_fromdn(), and eventually RSA_new(). The latter fails because libp11's RSA method does not have the flag RSA_FLAG_FIPS_METHOD set.

The bug can be easily reproduced on a FIPS system with OpenSSL command line tools.

Version-Release number of selected component (if applicable):
openssl-1.1.1d-2.fc31.x86_64
openssl-pkcs11-0.4.10-5.fc31.x86_64
softhsm-2.5.0-4.fc31.1.x86_64

How reproducible:
always

Steps to Reproduce:
1. put a system into FIPS mode (1minutetip --flavor ci.m1.medium.rng --fips Fedora-31)
2. Create /tmp/openssl.cnf with
[openssl_init]
engines=engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/lib64/pkcs11/libsofthsm2.so
MODULE_PATH = libsofthsm2.so
init = 0

3. OPENSSL_CONF=/tmp/openssl.cnf openssl genrsa -engine pkcs11


Actual results:
engine "pkcs11" set.
Generating RSA private key, 2048 bit long modulus (2 primes)
139740023297856:error:0406A0C8:rsa routines:RSA_new_method:non FIPS rsa method:crypto/rsa/rsa_lib.c:94:

Expected results:
no error

Additional info:

Traceback from Bind9 debug session:

(gdb) n
93          if (FIPS_mode() && !(ret->meth->flags & RSA_FLAG_FIPS_METHOD)) {
(gdb) n
94              RSAerr(RSA_F_RSA_NEW_METHOD, RSA_R_NON_FIPS_RSA_METHOD);

(gdb) bt
#0  RSA_new_method (engine=engine@entry=0x0) at crypto/rsa/rsa_lib.c:94
#1  0x00007ffff7b5575b in RSA_new () at crypto/rsa/rsa_lib.c:22
#2  0x00007ffff7f1be2f in opensslrsa_fromdns (key=0x7ffff4d462f0, data=0x7ffff71c7a80) at ../../../lib/dns/opensslrsa_link.c:1254
#3  0x00007ffff7f20042 in frombuffer (name=name@entry=0x7ffff71c7b00, alg=alg@entry=8, flags=flags@entry=257, protocol=protocol@entry=3, 
    rdclass=rdclass@entry=1, source=source@entry=0x7ffff71c7a80, mctx=0x5555556515a0, keyp=0x7ffff71c7968) at ../../../lib/dns/dst_api.c:1943
#4  0x00007ffff7f20240 in dst_key_fromdns (name=name@entry=0x7ffff71c7b00, rdclass=<optimized out>, source=source@entry=0x7ffff71c7a80, 
    mctx=mctx@entry=0x5555556515a0, keyp=keyp@entry=0x7ffff71c7a08) at ../../../lib/dns/dst_api.c:794
#5  0x00005555555ae41a in dstkey_fromconfig (vconfig=vconfig@entry=0x0, key=0x7ffff72083f8, managed=managed@entry=true, 
    target=target@entry=0x7ffff71c9d88, mctx=mctx@entry=0x5555556515a0) at ../../../bin/named/server.c:792
#6  0x00005555555ae592 in load_view_keys (keys=<optimized out>, vconfig=vconfig@entry=0x0, view=view@entry=0x7ffff00d2cf0, 
    managed=managed@entry=true, keyname=keyname@entry=0x0, mctx=mctx@entry=0x5555556515a0) at ../../../bin/named/server.c:845
#7  0x00005555555af09b in configure_view_dnsseckeys (view=view@entry=0x7ffff00d2cf0, vconfig=vconfig@entry=0x0, config=config@entry=0x7ffff72018a8, 
    bindkeys=bindkeys@entry=0x0, auto_root=auto_root@entry=false, mctx=mctx@entry=0x5555556515a0) at ../../../bin/named/server.c:1041
#8  0x00005555555bbdf6 in configure_view (view=<optimized out>, viewlist=viewlist@entry=0x7ffff71cc8c0, config=<optimized out>, 
    vconfig=vconfig@entry=0x0, cachelist=cachelist@entry=0x7ffff71cc8e0, bindkeys=<optimized out>, mctx=<optimized out>, actx=<optimized out>, 
    need_hints=<optimized out>) at ../../../bin/named/server.c:4705
#9  0x00005555555ca9ff in load_configuration (filename=<optimized out>, server=server@entry=0x7ffff71da010, first_time=first_time@entry=true)
    at ../../../bin/named/server.c:8125
#10 0x00005555555cbb02 in run_server (task=<optimized out>, event=<optimized out>) at ../../../bin/named/server.c:8887
#11 0x00007ffff7cbfebe in dispatch (manager=0x7ffff71d3010) at ../../../lib/isc/task.c:1145
#12 run (uap=0x7ffff71d3010) at ../../../lib/isc/task.c:1319
#13 0x00007ffff79444e2 in start_thread () from /lib64/libpthread.so.0
#14 0x00007ffff76e96a3 in clone () from /lib64/libc.so.6

(gdb) p *ret
$5 = {pad = 0, version = 0, meth = 0x555555631b10, engine = 0x5555556375d0, n = 0x0, e = 0x0, d = 0x0, p = 0x0, q = 0x0, dmp1 = 0x0, dmq1 = 0x0, 
  iqmp = 0x0, prime_infos = 0x0, pss = 0x0, ex_data = {sk = 0x0}, references = 1, flags = 0, _method_mod_n = 0x0, _method_mod_p = 0x0, 
  _method_mod_q = 0x0, bignum_data = 0x0, blinding = 0x0, mt_blinding = 0x0, lock = 0x7ffff0000b60}

(gdb) p *ret->engine
$6 = {id = 0x7ffff5199000 "pkcs11", name = 0x7ffff5199016 "pkcs11 engine", rsa_meth = 0x555555631b10, dsa_meth = 0x0, dh_meth = 0x0, 
  ec_meth = 0x555555631a90, rand_meth = 0x0, ciphers = 0x0, digests = 0x0, pkey_meths = 0x7ffff5196d80 <PKCS11_pkey_meths>, pkey_asn1_meths = 0x0, 
  destroy = 0x7ffff518e760 <engine_destroy>, init = 0x7ffff518e730 <engine_init>, finish = 0x7ffff518e700 <engine_finish>, 
  ctrl = 0x7ffff518e6b0 <engine_ctrl>, load_privkey = 0x7ffff518e600 <load_privkey>, load_pubkey = 0x7ffff518e670 <load_pubkey>, 
  load_ssl_client_cert = 0x0, cmd_defns = 0x7ffff519e720, flags = 0, struct_ref = 7, funct_ref = 5, ex_data = {sk = 0x5555556311e0}, 
  prev = 0x5555556376f0, next = 0x0

Comment 1 Christian Heimes 2020-04-24 07:26:00 UTC
PKCS11_get_rsa_method() does not set RSA_FLAG_FIPS_METHOD:

https://github.com/OpenSC/libp11/blob/4084f83ee5ea51353facf151126b7d6d739d0784/src/p11_rsa.c#L474-L493

IMHO the easiest solution is to provide a new CTRL that sets the appropriate FIPS method flags on engine's methods. This would permit users to claim FIPS compliance from an OpenSSL conf file.

In case it's possible to query the FIPS compliance state of a PKCS#11 module, then we can automate it, too. As far as I know PKCS#11 v2.x does not provide an API to query FIPS compliance of a provider. PKCS#11 v3 is finalized yet.

Comment 2 Christian Heimes 2020-04-24 08:01:41 UTC
Created attachment 1681390 [details]
Experimental FIPS hack

This hack resolves the problem for me. Bind9 named starts up and works with OpenSSL PKCS11 in FIPS mode. It's not a proper solution, though.

Comment 3 Christian Heimes 2020-04-27 13:16:05 UTC
Anderson Sasaki pointed out that the problem is fixed in RHEL 8.2 downstream.

Comment 4 Fedora Update System 2020-04-27 17:08:30 UTC
FEDORA-2020-85fcc9d518 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-85fcc9d518

Comment 5 Fedora Update System 2020-04-27 17:09:13 UTC
FEDORA-2020-8471ece45e has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-8471ece45e

Comment 6 Fedora Update System 2020-04-28 03:23:59 UTC
FEDORA-2020-8471ece45e has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-8471ece45e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-8471ece45e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Fedora Update System 2020-04-28 03:52:12 UTC
FEDORA-2020-85fcc9d518 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-85fcc9d518`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-85fcc9d518

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Christian Heimes 2020-04-28 06:56:27 UTC
The update solves the issue with Bind9 named and OpenSSL PKCS#11 engine in FIPS mode (tested on F31, F32 uses the same patch and version of libp11).

Comment 9 Fedora Update System 2020-05-01 00:36:23 UTC
FEDORA-2020-8471ece45e has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 10 Fedora Update System 2020-05-19 03:48:59 UTC
FEDORA-2020-85fcc9d518 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.