Bug 2050617

Summary: failed to initialize TLS context when fips=yes is used
Product: Red Hat Enterprise Linux 9 Reporter: Ondrej Moriš <omoris>
Component: stunnelAssignee: Clemens Lang <cllang>
Status: CLOSED ERRATA QA Contact: Ondrej Moriš <omoris>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 9.0Keywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: stunnel-5.62-2.el9 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 16:02:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ondrej Moriš 2022-02-04 10:25:50 UTC 
Description of problem:

When fips=yes is used in the configuration stunnels fails to start.

Version-Release number of selected component (if applicable):

stunnel-5.62-1.el9

How reproducible:

100%

Steps to Reproduce:

Configuration used:

output = /var/log/stunnel.log
cafile = /tmp/tmp.bvWkwKei7H/ca/cert.pem
cert = /tmp/tmp.bvWkwKei7H/server/cert.pem
key = /tmp/tmp.bvWkwKei7H/server/key.pem
fips = yes
debug = 7

[https]
accept = 20003
connect=127.0.0.1:80

Both CA and server keys are 3072 bit RSA keys.

1. /usr/bin/stunnel /etc/stunnel/stunnel.conf

Actual results:

[ ] Initializing inetd mode configuration
[ ] Clients allowed=500
[.] stunnel 5.62 on x86_64-redhat-linux-gnu platform
[.] Compiled with OpenSSL 3.0.0 7 sep 2021
[.] Running  with OpenSSL 3.0.1 14 Dec 2021
[.] Threading:PTHREAD Sockets:POLL,IPv6 TLS:ENGINE,FIPS,OCSP,PSK,SNI
[ ] errno: (*__errno_location ())
[ ] Initializing inetd mode configuration
[.] Reading configuration from file /etc/stunnel/stunnel.conf
[.] UTF-8 byte order mark not detected
[.] FIPS provider enabled
[.] FIPS mode enabled
[ ] Compression enabled: 0 methods
[ ] No PRNG seeding was required
[ ] Initializing service [https]
[ ] Using the default TLS version as specified in OpenSSL crypto policies. Not setting explicitly.
[ ] Using the default TLS version as specified in OpenSSL crypto policies. Not setting explicitly
[ ] OpenSSL security level is used: 2
[ ] Ciphers: FIPS
[ ] TLSv1.3 ciphersuites: TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256
[ ] TLS options: 0x2100000 (+0x0, -0x0)
[ ] Session resumption enabled
[ ] Loading certificate from file: /tmp/tmp.bvWkwKei7H/server/cert.pem
[ ] Certificate loaded from file: /tmp/tmp.bvWkwKei7H/server/cert.pem
[ ] Loading private key from file: /tmp/tmp.bvWkwKei7H/server/key.pem
[ ] Private key loaded from file: /tmp/tmp.bvWkwKei7H/server/key.pem
[ ] Private key check succeeded
[ ] Client CA list: /tmp/tmp.bvWkwKei7H/ca/cert.pem
[ ] Client CA: O=Example CA
[ ] DH initialization needed for DHE-RSA-AES256-GCM-SHA384
[ ] DH initialization
[ ] Could not load DH parameters from /tmp/tmp.bvWkwKei7H/server/cert.pem
[ ] Using dynamic DH parameters
[ ] ECDH initialization
[!] Invalid groups list in 'curves'
[!] Service [https]: Failed to initialize TLS context
[!] Configuration failed
[ ] Deallocating temporary section defaults
[ ] Deallocating section [https]

Exited with 1.

Expected results:

Exit with 0, initialize TLS context.

Additional info:

System is not in FIPS mode. FYI, the last stunnel version working with fips=yes was 5.48 (RHEL-8.2). However, failures were different than the one reported here - stunnel started correctly but connection were not working.
Comment 1 Ondrej Moriš 2022-02-04 10:51:12 UTC 
FYI, results are the same when FIPS mode is enabled on the system.

On RHEL-8 errors on the server (running stunnel on 20003 and connecting to apache) were as follows after client connection (client runs stunnel on 80 connecting to servers 20003):

2022.02.04 05:48:19 LOG7[ui]: Clients allowed=500
2022.02.04 05:48:19 LOG5[ui]: stunnel 5.56 on x86_64-redhat-linux-gnu platform
2022.02.04 05:48:19 LOG5[ui]: Compiled with OpenSSL 1.1.1g FIPS  21 Apr 2020
2022.02.04 05:48:19 LOG5[ui]: Running  with OpenSSL 1.1.1k  FIPS 25 Mar 2021
2022.02.04 05:48:19 LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6 TLS:ENGINE,FIPS,OCSP,PSK,SNI
2022.02.04 05:48:19 LOG7[ui]: errno: (*__errno_location ())
2022.02.04 05:48:19 LOG5[ui]: Reading configuration from file /etc/stunnel/stunnel.conf
2022.02.04 05:48:19 LOG5[ui]: UTF-8 byte order mark not detected
2022.02.04 05:48:19 LOG5[ui]: FIPS mode enabled
2022.02.04 05:48:19 LOG7[ui]: Compression disabled
2022.02.04 05:48:19 LOG7[ui]: No PRNG seeding was required
2022.02.04 05:48:19 LOG6[ui]: Initializing service [https]
2022.02.04 05:48:19 LOG6[ui]: Using the default TLS version as specified in                 OpenSSL crypto policies. Not setting explicitly.
2022.02.04 05:48:19 LOG6[ui]: Using the default TLS version as specified in                 OpenSSL crypto policies. Not setting explicitly
2022.02.04 05:48:19 LOG7[ui]: Ciphers: FIPS
2022.02.04 05:48:19 LOG7[ui]: TLSv1.3 ciphersuites: TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256
2022.02.04 05:48:19 LOG7[ui]: TLS options: 0x02100004 (+0x00000000, -0x00000000)
2022.02.04 05:48:19 LOG6[ui]: Loading certificate from file: /tmp/tmp.fKfzywKlkx/server/cert.pem
2022.02.04 05:48:19 LOG6[ui]: Certificate loaded from file: /tmp/tmp.fKfzywKlkx/server/cert.pem
2022.02.04 05:48:19 LOG6[ui]: Loading private key from file: /tmp/tmp.fKfzywKlkx/server/key.pem
2022.02.04 05:48:19 LOG6[ui]: Private key loaded from file: /tmp/tmp.fKfzywKlkx/server/key.pem
2022.02.04 05:48:19 LOG7[ui]: Private key check succeeded
2022.02.04 05:48:19 LOG7[ui]: Client CA list: /tmp/tmp.fKfzywKlkx/ca/cert.pem
2022.02.04 05:48:19 LOG6[ui]: Client CA: O=Example CA
2022.02.04 05:48:19 LOG6[ui]: DH initialization needed for DHE-DSS-AES256-GCM-SHA384
2022.02.04 05:48:19 LOG7[ui]: DH initialization
2022.02.04 05:48:19 LOG7[ui]: Could not load DH parameters from /tmp/tmp.fKfzywKlkx/server/cert.pem
2022.02.04 05:48:19 LOG6[ui]: Using dynamic DH parameters
2022.02.04 05:48:19 LOG7[ui]: ECDH initialization
2022.02.04 05:48:19 LOG7[ui]: ECDH initialized with curves X25519:P-256:X448:P-521:P-384
2022.02.04 05:48:19 LOG5[ui]: Configuration successful
2022.02.04 05:48:19 LOG7[ui]: Binding service [https]
2022.02.04 05:48:19 LOG7[ui]: Listening file descriptor created (FD=9)
2022.02.04 05:48:19 LOG7[ui]: Setting accept socket options (FD=9)
2022.02.04 05:48:19 LOG7[ui]: Option SO_REUSEADDR set on accept socket
2022.02.04 05:48:19 LOG6[ui]: Service [https] (FD=9) bound to 0.0.0.0:20003
2022.02.04 05:48:19 LOG7[ui]: Listening file descriptor created (FD=10)
2022.02.04 05:48:19 LOG7[ui]: Setting accept socket options (FD=10)
2022.02.04 05:48:19 LOG7[ui]: Option SO_REUSEADDR set on accept socket
2022.02.04 05:48:19 LOG5[ui]: Binding service [https] to :::20003: Address already in use (98)
2022.02.04 05:48:19 LOG7[main]: No pid file being created
2022.02.04 05:48:19 LOG7[cron]: Cron thread initialized
2022.02.04 05:48:19 LOG6[cron]: Executing cron jobs
2022.02.04 05:48:19 LOG5[cron]: Updating DH parameters
2022.02.04 05:48:19 LOG3[cron]: DH_generate_parameters_ex: crypto/dh/dh_gen.c:31: error:050C90CA:Diffie-Hellman routines:DH_generate_parameters_ex:non FIPS method
2022.02.04 05:48:19 LOG6[cron]: Cron jobs completed in 0 seconds
2022.02.04 05:48:19 LOG7[cron]: Waiting 86400 seconds
2022.02.04 05:48:43 LOG7[main]: Found 1 ready file descriptor(s)
2022.02.04 05:48:43 LOG7[main]: FD=4 events=0x2001 revents=0x0
2022.02.04 05:48:43 LOG7[main]: FD=9 events=0x2001 revents=0x1
2022.02.04 05:48:43 LOG7[main]: Service [https] accepted (FD=3) from 10.0.139.100:57696
2022.02.04 05:48:43 LOG7[0]: Service [https] started
2022.02.04 05:48:43 LOG7[0]: Setting local socket options (FD=3)
2022.02.04 05:48:43 LOG7[0]: Option TCP_NODELAY set on local socket
2022.02.04 05:48:43 LOG5[0]: Service [https] accepted connection from 10.0.139.100:57696
2022.02.04 05:48:43 LOG6[0]: Peer certificate not required
2022.02.04 05:48:43 LOG7[0]: TLS state (accept): before SSL initialization
2022.02.04 05:48:43 LOG7[0]: TLS state (accept): before SSL initialization
2022.02.04 05:48:43 LOG7[0]: Initializing application specific data for session authenticated
2022.02.04 05:48:43 LOG7[0]: SNI: no virtual services defined
2022.02.04 05:48:43 LOG7[0]: TLS state (accept): SSLv3/TLS read client hello
2022.02.04 05:48:43 LOG7[0]: TLS state (accept): SSLv3/TLS write server hello
2022.02.04 05:48:43 LOG3[0]: error queue: ssl/tls13_enc.c:427: error:14202006:SSL routines:derive_secret_key_and_iv:EVP lib
2022.02.04 05:48:43 LOG3[0]: SSL_accept: crypto/evp/evp_enc.c:227: error:0607B0C8:digital envelope routines:EVP_CipherInit_ex:disabled for FIPS

(stunnel-5.56-5.el8_3)
Comment 2 Clemens Lang 2022-02-04 11:45:37 UTC 
I have reproduced the problem. stunnel defaults to calling SSL_CTX_set1_groups_list with X25519:P-256:X448:P-521:P-384, but in FIPS mode X25519 and X448 do not work.
We should probably remove them from the default when FIPS mode is enabled.
Comment 3 Clemens Lang 2022-02-04 15:16:31 UTC 
https://src.fedoraproject.org/rpms/stunnel/pull-request/8
https://gitlab.com/redhat/centos-stream/rpms/stunnel/-/merge_requests/6
Comment 4 Ondrej Moriš 2022-02-04 15:49:07 UTC 
Acceptance Criteria:

Suppose we have a client and a server. The server is running stunnel on port 20003 connecting to local http server running on port 80. The client is running stunnel on port 80 connecting to server's stunnel port.

 * [no gating] curl from the client connects to the server http in (stunnel) FIPS mode.

Please notice that is already implemented in (multhost) TC#444421.
Comment 8 Clemens Lang 2022-02-07 15:09:53 UTC 
Verified:

# cat stunnel.conf
fips = yes
debug = 6
cafile = keys/ca/cert.pem
output = /tmp/stunnel.log

[server]
accept = 20003
connect = 127.0.0.1:8080
cert = keys/server/cert.pem
key = keys/server/key.pem
verifyChain = yes

[client]
client = yes
accept = 80
connect = 127.0.0.1:20003
cert = keys/client/cert.pem
key = keys/client/key.pem


# python3 -mhttp.server 8080 &
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...
[1] 45

# ./src/stunnel stunnel.conf
stunnel: LOG6[ui]: Initializing inetd mode configuration
stunnel: LOG5[ui]: stunnel 5.62 on x86_64-koji-linux-gnu platform
stunnel: LOG5[ui]: Compiled/running with OpenSSL 3.0.1 14 Dec 2021
stunnel: LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6 TLS:ENGINE,FIPS,OCSP,PSK,SNI
stunnel: LOG6[ui]: Initializing inetd mode configuration
stunnel: LOG5[ui]: Reading configuration from file /builddir/build/BUILD/stunnel-5.62/stunnel.conf
stunnel: LOG5[ui]: UTF-8 byte order mark not detected
stunnel: LOG5[ui]: FIPS provider enabled
stunnel: LOG5[ui]: FIPS mode enabled
stunnel: LOG6[ui]: Compression enabled: 0 methods
stunnel: LOG6[ui]: Initializing service [server]
stunnel: LOG6[ui]: Using the default TLS version as specified in OpenSSL crypto policies. Not setting explicitly.
stunnel: LOG6[ui]: Using the default TLS version as specified in OpenSSL crypto policies. Not setting explicitly
stunnel: LOG6[ui]: OpenSSL security level is used: 2
stunnel: LOG6[ui]: Session resumption enabled
stunnel: LOG6[ui]: Loading certificate from file: keys/server/cert.pem
stunnel: LOG6[ui]: Certificate loaded from file: keys/server/cert.pem
stunnel: LOG6[ui]: Loading private key from file: keys/server/key.pem
stunnel: LOG6[ui]: Private key loaded from file: keys/server/key.pem
stunnel: LOG6[ui]: Client CA: O=Example CA
stunnel: LOG6[ui]: DH initialization needed for DHE-RSA-AES256-GCM-SHA384
stunnel: LOG6[ui]: Using dynamic DH parameters
stunnel: LOG6[ui]: Initializing service [client]
stunnel: LOG6[ui]: Using the default TLS version as specified in OpenSSL crypto policies. Not setting explicitly.
stunnel: LOG6[ui]: Using the default TLS version as specified in OpenSSL crypto policies. Not setting explicitly
stunnel: LOG6[ui]: OpenSSL security level is used: 2
stunnel: LOG6[ui]: Session resumption enabled
stunnel: LOG6[ui]: Loading certificate from file: keys/client/cert.pem
stunnel: LOG6[ui]: Certificate loaded from file: keys/client/cert.pem
stunnel: LOG6[ui]: Loading private key from file: keys/client/key.pem
stunnel: LOG6[ui]: Private key loaded from file: keys/client/key.pem
stunnel: LOG4[ui]: Service [client] needs authentication to prevent MITM attacks
stunnel: LOG6[ui]: DH initialization skipped: client section
stunnel: LOG5[ui]: Configuration successful
stunnel: LOG6[ui]: Service [server] (FD=8) bound to 0.0.0.0:20003
stunnel: LOG5[ui]: Binding service [server] to :::20003: Address already in use (98)
stunnel: LOG6[ui]: Service [client] (FD=9) bound to 0.0.0.0:80
stunnel: LOG5[ui]: Binding service [client] to :::80: Address already in use (98)
stunnel: LOG6[cron]: Executing cron jobs
stunnel: LOG5[cron]: Updating DH parameters

# curl -IL http://127.0.0.1/
stunnel: LOG5[0]: Service [client] accepted connection from 127.0.0.1:34422
stunnel: LOG6[0]: s_connect: connecting 127.0.0.1:20003
stunnel: LOG5[0]: s_connect: connected 127.0.0.1:20003
stunnel: LOG5[0]: Service [client] connected remote server from 127.0.0.1:43420
stunnel: LOG5[1]: Service [server] accepted connection from 127.0.0.1:43420
stunnel: LOG6[0]: SNI: sending servername: 127.0.0.1
stunnel: LOG6[0]: Peer certificate not required
stunnel: LOG6[1]: Peer certificate required
stunnel: LOG6[0]: Client CA: O=Example CA
stunnel: LOG6[0]: Certificate verification disabled
stunnel: LOG6[0]: Certificate verification disabled
stunnel: LOG6[0]: TLS connected: new session negotiated
stunnel: LOG6[0]: TLSv1.3 ciphersuite: TLS_AES_256_GCM_SHA384 (256-bit encryption)
stunnel: LOG6[0]: Peer temporary key: ECDH, P-256, 256 bits
stunnel: LOG6[1]: Certificate accepted at depth=1: O=Example CA
stunnel: LOG6[1]: CERT: No subject checks configured
stunnel: LOG5[1]: Certificate accepted at depth=0: CN=John Smith
stunnel: LOG6[1]: Session id: 3F406EF5B09DDCC3CC14BCC4054157760C99E3D62A4223EA40230C2B2AAAEB49
stunnel: LOG6[0]: Session id: DF4D3798786DE3E3F30D532726E36B7946ADABB10E232673E2CA70226E73A14D
stunnel: LOG6[1]: Session id: 16B7EDFC788F9BCB30E24693C6EEEDCA5B48B5B490C55F9A709FF1E02ED41D6B
stunnel: LOG6[1]: TLS accepted: new session negotiated
stunnel: LOG6[1]: TLSv1.3 ciphersuite: TLS_AES_256_GCM_SHA384 (256-bit encryption)
stunnel: LOG6[0]: Session id: 8955499037677CD389A4489144DA345B59B2263E0C82ED6D30C1D463D2A3C9F0
stunnel: LOG6[1]: Peer temporary key: ECDH, P-256, 256 bits
stunnel: LOG6[1]: s_connect: connecting 127.0.0.1:8080
stunnel: LOG5[1]: s_connect: connected 127.0.0.1:8080
stunnel: LOG6[1]: persistence: 127.0.0.1:8080 cached
stunnel: LOG5[1]: Service [server] connected remote server from 127.0.0.1:42496
127.0.0.1 - - [07/Feb/2022 16:06:30] "HEAD / HTTP/1.1" 200 -
HTTP/1.0 200 OK
stunnel: LOG6[1]: Read socket closed (readsocket)
Server: SimpleHTTP/0.6 Python/3.9.10
Date: Mon, 07 Feb 2022 15:06:30 GMT
Content-type: text/html; charset=utf-8
Content-Length: 2033

stunnel: LOG6[1]: SSL_shutdown successfully sent close_notify alert
stunnel: LOG6[0]: Read socket closed (readsocket)
stunnel: LOG6[0]: TLS closed (SSL_read)
stunnel: LOG6[0]: SSL_shutdown successfully sent close_notify alert
stunnel: LOG6[1]: TLS closed (SSL_read)
stunnel: LOG5[0]: Connection closed: 74 byte(s) sent to TLS, 156 byte(s) sent to socket
stunnel: LOG5[1]: Connection closed: 156 byte(s) sent to TLS, 74 byte(s) sent to socket
Comment 13 errata-xmlrpc 2022-05-17 16:02:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: stunnel), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4036