Description of problem: ------------------------ With the default tlsSecurityProfile (intermediate profile), connection validation for the components: virt-api, cdi-api, ssp-operator-service, ssp-operator-metrics, hostpath-provisioner-operator-service and kube-macpool-service, fails for TLS v1.2 sh-5.1# openssl s_client -connect 172.30.29.99:443 -tls1_3 -brief <<< 'Q' 2>&1 | grep 'Protocol version' Protocol version: TLSv1.3 sh-5.1# openssl s_client -connect 172.30.29.99:443 -tls1_2 -brief <<< 'Q' 2>&1 | grep 'Protocol version' sh-5.1# openssl s_client -connect 172.30.29.99:443 -tls1_1 -brief <<< 'Q' 2>&1 | grep 'Protocol version' sh-5.1# sh-5.1# openssl s_client -connect 172.30.29.99:443 -tls1_2 -brief <<< 'Q' 2>&1 Can't use SSL_get_servername depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=20:unable to get local issuer certificate depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=21:unable to verify the first certificate 805BCA1FBB7F0000:error:1C8000E9:Provider routines:kdf_tls1_prf_derive:ems not enabled:providers/implementations/kdfs/tls1_prf.c:194: 805BCA1FBB7F0000:error:0A08010C:SSL routines:tls1_PRF:unsupported:ssl/t1_enc.c:83: Version-Release number of selected component (if applicable): -------------------------------------------------------------- CNV v4.13.3 & CNV 4.14 How reproducible: ----------------- Always Steps to Reproduce: ------------------- 0. Create a 6 node cluster ( 3 master + 3 worker ) with FIPS enabled. 1. Get the IP & port number of a service 'virt-api' # oc get service -n openshift-cnv | grep virt-api virt-api ClusterIP 172.30.29.99 <none> 443/TCP 17h2. 2. Validate the connection from one of the worker node # oc debug node/<worker-node> # chroot /host # openssl s_client -connect 172.30.29.99:443 -tls1_2 -brief <<< 'Q' 2>&1 Actual results: --------------- Error when validating the connection with TLS v1.2 sh-5.1# openssl s_client -connect 172.30.29.99:443 -tls1_2 -brief <<< 'Q' 2>&1 Can't use SSL_get_servername depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=20:unable to get local issuer certificate depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=21:unable to verify the first certificate 801BD9D4EC7F0000:error:1C8000E9:Provider routines:kdf_tls1_prf_derive:ems not enabled:providers/implementations/kdfs/tls1_prf.c:194: 801BD9D4EC7F0000:error:0A08010C:SSL routines:tls1_PRF:unsupported:ssl/t1_enc.c:83: Expected results: ----------------- Connection validation should be successful with TLS v1.2, like the same way it happens for TLS v1.3 sh-5.1# openssl s_client -connect 172.30.29.99:443 -tls1_3 -brief <<< 'Q' 2>&1 Can't use SSL_get_servername depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=20:unable to get local issuer certificate depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=21:unable to verify the first certificate CONNECTION ESTABLISHED Protocol version: TLSv1.3 Ciphersuite: TLS_AES_128_GCM_SHA256 Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512 Peer certificate: CN = virt-api.openshift-cnv.pod.cluster.local Hash used: SHA256 Signature type: ECDSA Verification error: unable to verify the first certificate Server Temp Key: ECDH, prime256v1, 256 bits DONE Additional info: ----------------- This issue is also found with CNV v4.14
When FIPs is disabled in CNV v4.13.3, the connection could be validated for TLS v1.2 sh-5.1# openssl s_client --connect 172.30.193.135:443 -tls1_2 -brief <<< 'Q' 2>&1 | grep 'Protocol version' Protocol version: TLSv1.2 sh-5.1# openssl s_client --connect 172.30.193.135:443 -tls1_3 -brief <<< 'Q' 2>&1 | grep 'Protocol version' Protocol version: TLSv1.3
If I expose that service with: oc port-forward -n openshift-cnv service/virt-api 1443:443 & And then I try to connect with the openssl client on my laptop I get: $ openssl s_client --connect 127.0.0.1:1443 -tls1_2 -brief <<< 'Q' 2>&1 Handling connection for 1443 Can't use SSL_get_servername depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=20:unable to get local issuer certificate depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=21:unable to verify the first certificate CONNECTION ESTABLISHED Protocol version: TLSv1.2 Ciphersuite: ECDHE-ECDSA-AES128-GCM-SHA256 Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512 Peer certificate: CN = virt-api.openshift-cnv.pod.cluster.local Hash used: SHA256 Signature type: ECDSA Verification error: unable to verify the first certificate Supported Elliptic Curve Point Formats: uncompressed Server Temp Key: ECDH, prime256v1, 256 bits DONE So it choosed Ciphersuite: ECDHE-ECDSA-AES128-GCM-SHA256 on Protocol version: TLSv1.2. But if I check on one of worker nodes: $ oc debug node/dev-simone-1-jzscx-worker-0-pb5fj Temporary namespace openshift-debug-4cgbr is created for debugging node... Starting pod/dev-simone-1-jzscx-worker-0-pb5fj-debug ... To use host binaries, run `chroot /host` Pod IP: 192.168.2.216 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-5.1# openssl ciphers TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-CCM:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-CCM:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-CCM:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-CCM:PSK-AES256-GCM-SHA384:PSK-AES256-CCM:PSK-AES128-GCM-SHA256:PSK-AES128-CCM:DHE-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-CCM:DHE-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-CCM sh-5.1# openssl ciphers FIPS TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ADH-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ADH-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ADH-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ADH-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:AECDH-AES256-SHA:ADH-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:AECDH-AES128-SHA:ADH-AES128-SHA:DHE-PSK-AES256-GCM-SHA384:PSK-AES256-GCM-SHA384:DHE-PSK-AES128-GCM-SHA256:PSK-AES128-GCM-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA sh-5.1# openssl s_client -showcerts -connect 172.30.212.197:443 -tls1_2 <<< 'Q' 2>&1 CONNECTED(00000003) Can't use SSL_get_servername depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify error:num=21:unable to verify the first certificate verify return:1 depth=0 CN = virt-api.openshift-cnv.pod.cluster.local verify return:1 408CDE0E927F0000:error:1C8000E9:Provider routines:kdf_tls1_prf_derive:ems not enabled:providers/implementations/kdfs/tls1_prf.c:194: 408CDE0E927F0000:error:0A08010C:SSL routines:tls1_PRF:unsupported:ssl/t1_enc.c:83: --- Certificate chain 0 s:CN = virt-api.openshift-cnv.pod.cluster.local i:CN = kubevirt.io@1689853842 a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Jul 20 11:50:42 2023 GMT; NotAfter: Jul 21 11:50:42 2023 GMT -----BEGIN CERTIFICATE----- MIICBjCCAa2gAwIBAgIIHYG7Jq0gfzgwCgYIKoZIzj0EAwIwITEfMB0GA1UEAwwW a3ViZXZpcnQuaW9AMTY4OTg1Mzg0MjAeFw0yMzA3MjAxMTUwNDJaFw0yMzA3MjEx MTUwNDJaMDMxMTAvBgNVBAMTKHZpcnQtYXBpLm9wZW5zaGlmdC1jbnYucG9kLmNs dXN0ZXIubG9jYWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATBkjrQH2JQpnqr UWHUe1SAVOxcmZIxypexYLT/acrX5fJCBImxvN2XAE+ZgSxB5zKckAEZhtCmiqqI 92nOG4RXo4G8MIG5MA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcD ATAfBgNVHSMEGDAWgBT7xl8ORnRiI8uzU9kwOtDylwfnFjBxBgNVHREEajBoggh2 aXJ0LWFwaYIWdmlydC1hcGkub3BlbnNoaWZ0LWNudoIadmlydC1hcGkub3BlbnNo aWZ0LWNudi5zdmOCKHZpcnQtYXBpLm9wZW5zaGlmdC1jbnYuc3ZjLmNsdXN0ZXIu bG9jYWwwCgYIKoZIzj0EAwIDRwAwRAIge19AxksPhDwyS7XSwnKUg8jrFksnsqZ1 xQ/MmQtXWqACIB6dyfET8bHXYtRBfnOb5739g+10x+Bu2M57o01UK6KH -----END CERTIFICATE----- --- Server certificate subject=CN = virt-api.openshift-cnv.pod.cluster.local issuer=CN = kubevirt.io@1689853842 --- Acceptable client certificate CA names CN = openshift-kube-apiserver-operator_aggregator-client-signer@1689827951 CN = openshift-kube-apiserver-operator_aggregator-client-signer@1689849551 Client Certificate Types: RSA sign, ECDSA sign Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512 Shared Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512 Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: ECDH, prime256v1, 256 bits --- SSL handshake has read 965 bytes and written 232 bytes Verification error: unable to verify the first certificate --- New, (NONE), Cipher is (NONE) Server public key is 256 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: PSK identity: None PSK identity hint: None SRP username: None Start Time: 1689853946 Timeout : 7200 (sec) Verify return code: 21 (unable to verify the first certificate) Extended master secret: no --- sh-5.1#
On the worker node we have: sh-5.1# openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022) On my laptop: $ openssl version OpenSSL 3.0.9 30 May 2023 (Library: OpenSSL 3.0.9 30 May 2023)
The actual error is: 408CDE0E927F0000:error:1C8000E9:Provider routines:kdf_tls1_prf_derive:ems not enabled:providers/implementations/kdfs/tls1_prf.c:194: 408CDE0E927F0000:error:0A08010C:SSL routines:tls1_PRF:unsupported:ssl/t1_enc.c:83: and in https://github.com/openssl/openssl/blob/master/CHANGES.md#changes-between-30-and-310-14-mar-2023 we have: Changes between 3.0 and 3.1.0 [14 Mar 2023] Add FIPS provider configuration option to enforce the Extended Master Secret (EMS) check during the TLS1_PRF KDF. The option '-ems-check' can optionally be supplied to 'openssl fipsinstall'.
On our container images we currently have: bash-5.1$ rpm -qa | grep openssl openssl-libs-3.0.7-6.el9_2.x86_64 openssl-3.0.7-6.el9_2.x86_64 while on the RHCOS 9.2 nodes we have: > rpm -qa 'openssl*' openssl-libs-3.0.7-16.el9_2.x86_64 openssl-3.0.7-16.el9_2.x86_64 and the fix for https://bugzilla.redhat.com/show_bug.cgi?id=2157951 landed in openssl-3.0.7-16.el9 See also: https://bugzilla.redhat.com/show_bug.cgi?id=2188046
See also: https://access.redhat.com/solutions/7018256 - TLS Extension "Extended Master Secret" enforced with Red Hat Enterprise Linux 9.2
workaround, when in FIPS mode configure CNV to require the modern TLS security profile: oc patch hco -n openshift-cnv --type=json kubevirt-hyperconverged -p '[{"op": "replace", "path": /spec/tlsSecurityProfile, "value": {modern: {}, type: "Modern"} }]' See: stirabos@t14s:~$ oc patch hco -n openshift-cnv --type=json kubevirt-hyperconverged -p '[{"op": "replace", "path": /spec/tlsSecurityProfile, "value": {modern: {}, type: "Modern"} }]' hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched stirabos@t14s:~$ oc debug node/dev-simone-1-jzscx-worker-0-pb5fj Temporary namespace openshift-debug-6cvdp is created for debugging node... Starting pod/dev-simone-1-jzscx-worker-0-pb5fj-debug ... To use host binaries, run `chroot /host` Pod IP: 192.168.2.216 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-5.1# openssl s_client -showcerts -connect 172.30.212.197:443 -tls1_2 <<< 'Q' 2>&1 CONNECTED(00000003) 40ECA38C5B7F0000:error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1600:SSL alert number 70 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 138 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: PSK identity: None PSK identity hint: None SRP username: None Start Time: 1689868773 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no --- sh-5.1# exit exit sh-4.4# exit exit Removing debug pod ... Temporary namespace openshift-debug-6cvdp was removed. stirabos@t14s:~$
EMS implementation landed in golang only at the end of may, see: https://go-review.googlesource.com/c/go/+/497376 and it's not realistically going to be backported to golang 1.20 (used to build CNV 4.14) not golang 1.19 (used to build CNV 4.13) so we don't have any realistic option to get this properly fixed before CNV 4.15. Closing as WONTFIX. The best we can do is suggesting to our customers to configure CNV with the modern tlsSecurity profile when in FIPS mode, this will force the server to negotiate only TLS 1.3 emitting a clear error on obsolete clients. Thew work for the release note is stracked in https://issues.redhat.com/browse/CNV-31344