Bug 2224204
| Summary: | [crypto-policy] Connection validation fails for TLS v1.2 with the default tlsSecurityProfile | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | SATHEESARAN <sasundar> |
| Component: | Installation | Assignee: | Simone Tiraboschi <stirabos> |
| Status: | CLOSED WONTFIX | QA Contact: | SATHEESARAN <sasundar> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.13.3 | CC: | dholler, dshchedr, fdeutsch, stirabos |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | 4.13.3 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-07-25 12:55:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
SATHEESARAN
2023-07-20 07:51:17 UTC
When FIPs is disabled in CNV v4.13.3, the connection could be validated for TLS v1.2 sh-5.1# openssl s_client --connect 172.30.193.135:443 -tls1_2 -brief <<< 'Q' 2>&1 | grep 'Protocol version' Protocol version: TLSv1.2 sh-5.1# openssl s_client --connect 172.30.193.135:443 -tls1_3 -brief <<< 'Q' 2>&1 | grep 'Protocol version' Protocol version: TLSv1.3 If I expose that service with:
oc port-forward -n openshift-cnv service/virt-api 1443:443 &
And then I try to connect with the openssl client on my laptop I get:
$ openssl s_client --connect 127.0.0.1:1443 -tls1_2 -brief <<< 'Q' 2>&1
Handling connection for 1443
Can't use SSL_get_servername
depth=0 CN = virt-api.openshift-cnv.pod.cluster.local
verify error:num=20:unable to get local issuer certificate
depth=0 CN = virt-api.openshift-cnv.pod.cluster.local
verify error:num=21:unable to verify the first certificate
CONNECTION ESTABLISHED
Protocol version: TLSv1.2
Ciphersuite: ECDHE-ECDSA-AES128-GCM-SHA256
Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512
Peer certificate: CN = virt-api.openshift-cnv.pod.cluster.local
Hash used: SHA256
Signature type: ECDSA
Verification error: unable to verify the first certificate
Supported Elliptic Curve Point Formats: uncompressed
Server Temp Key: ECDH, prime256v1, 256 bits
DONE
So it choosed Ciphersuite: ECDHE-ECDSA-AES128-GCM-SHA256 on Protocol version: TLSv1.2.
But if I check on one of worker nodes:
$ oc debug node/dev-simone-1-jzscx-worker-0-pb5fj
Temporary namespace openshift-debug-4cgbr is created for debugging node...
Starting pod/dev-simone-1-jzscx-worker-0-pb5fj-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.2.216
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-5.1# openssl ciphers
TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-CCM:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-CCM:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-CCM:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-CCM:PSK-AES256-GCM-SHA384:PSK-AES256-CCM:PSK-AES128-GCM-SHA256:PSK-AES128-CCM:DHE-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-CCM:DHE-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-CCM
sh-5.1# openssl ciphers FIPS
TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ADH-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ADH-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ADH-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ADH-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:AECDH-AES256-SHA:ADH-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:AECDH-AES128-SHA:ADH-AES128-SHA:DHE-PSK-AES256-GCM-SHA384:PSK-AES256-GCM-SHA384:DHE-PSK-AES128-GCM-SHA256:PSK-AES128-GCM-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA
sh-5.1# openssl s_client -showcerts -connect 172.30.212.197:443 -tls1_2 <<< 'Q' 2>&1
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = virt-api.openshift-cnv.pod.cluster.local
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = virt-api.openshift-cnv.pod.cluster.local
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = virt-api.openshift-cnv.pod.cluster.local
verify return:1
408CDE0E927F0000:error:1C8000E9:Provider routines:kdf_tls1_prf_derive:ems not enabled:providers/implementations/kdfs/tls1_prf.c:194:
408CDE0E927F0000:error:0A08010C:SSL routines:tls1_PRF:unsupported:ssl/t1_enc.c:83:
---
Certificate chain
0 s:CN = virt-api.openshift-cnv.pod.cluster.local
i:CN = kubevirt.io@1689853842
a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
v:NotBefore: Jul 20 11:50:42 2023 GMT; NotAfter: Jul 21 11:50:42 2023 GMT
-----BEGIN CERTIFICATE-----
MIICBjCCAa2gAwIBAgIIHYG7Jq0gfzgwCgYIKoZIzj0EAwIwITEfMB0GA1UEAwwW
a3ViZXZpcnQuaW9AMTY4OTg1Mzg0MjAeFw0yMzA3MjAxMTUwNDJaFw0yMzA3MjEx
MTUwNDJaMDMxMTAvBgNVBAMTKHZpcnQtYXBpLm9wZW5zaGlmdC1jbnYucG9kLmNs
dXN0ZXIubG9jYWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATBkjrQH2JQpnqr
UWHUe1SAVOxcmZIxypexYLT/acrX5fJCBImxvN2XAE+ZgSxB5zKckAEZhtCmiqqI
92nOG4RXo4G8MIG5MA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcD
ATAfBgNVHSMEGDAWgBT7xl8ORnRiI8uzU9kwOtDylwfnFjBxBgNVHREEajBoggh2
aXJ0LWFwaYIWdmlydC1hcGkub3BlbnNoaWZ0LWNudoIadmlydC1hcGkub3BlbnNo
aWZ0LWNudi5zdmOCKHZpcnQtYXBpLm9wZW5zaGlmdC1jbnYuc3ZjLmNsdXN0ZXIu
bG9jYWwwCgYIKoZIzj0EAwIDRwAwRAIge19AxksPhDwyS7XSwnKUg8jrFksnsqZ1
xQ/MmQtXWqACIB6dyfET8bHXYtRBfnOb5739g+10x+Bu2M57o01UK6KH
-----END CERTIFICATE-----
---
Server certificate
subject=CN = virt-api.openshift-cnv.pod.cluster.local
issuer=CN = kubevirt.io@1689853842
---
Acceptable client certificate CA names
CN = openshift-kube-apiserver-operator_aggregator-client-signer@1689827951
CN = openshift-kube-apiserver-operator_aggregator-client-signer@1689849551
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512
Shared Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 965 bytes and written 232 bytes
Verification error: unable to verify the first certificate
---
New, (NONE), Cipher is (NONE)
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1689853946
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: no
---
sh-5.1#
On the worker node we have: sh-5.1# openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022) On my laptop: $ openssl version OpenSSL 3.0.9 30 May 2023 (Library: OpenSSL 3.0.9 30 May 2023) The actual error is: 408CDE0E927F0000:error:1C8000E9:Provider routines:kdf_tls1_prf_derive:ems not enabled:providers/implementations/kdfs/tls1_prf.c:194: 408CDE0E927F0000:error:0A08010C:SSL routines:tls1_PRF:unsupported:ssl/t1_enc.c:83: and in https://github.com/openssl/openssl/blob/master/CHANGES.md#changes-between-30-and-310-14-mar-2023 we have: Changes between 3.0 and 3.1.0 [14 Mar 2023] Add FIPS provider configuration option to enforce the Extended Master Secret (EMS) check during the TLS1_PRF KDF. The option '-ems-check' can optionally be supplied to 'openssl fipsinstall'. On our container images we currently have: bash-5.1$ rpm -qa | grep openssl openssl-libs-3.0.7-6.el9_2.x86_64 openssl-3.0.7-6.el9_2.x86_64 while on the RHCOS 9.2 nodes we have: > rpm -qa 'openssl*' openssl-libs-3.0.7-16.el9_2.x86_64 openssl-3.0.7-16.el9_2.x86_64 and the fix for https://bugzilla.redhat.com/show_bug.cgi?id=2157951 landed in openssl-3.0.7-16.el9 See also: https://bugzilla.redhat.com/show_bug.cgi?id=2188046 See also: https://access.redhat.com/solutions/7018256 - TLS Extension "Extended Master Secret" enforced with Red Hat Enterprise Linux 9.2 workaround, when in FIPS mode configure CNV to require the modern TLS security profile:
oc patch hco -n openshift-cnv --type=json kubevirt-hyperconverged -p '[{"op": "replace", "path": /spec/tlsSecurityProfile, "value": {modern: {}, type: "Modern"} }]'
See:
stirabos@t14s:~$ oc patch hco -n openshift-cnv --type=json kubevirt-hyperconverged -p '[{"op": "replace", "path": /spec/tlsSecurityProfile, "value": {modern: {}, type: "Modern"} }]'
hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched
stirabos@t14s:~$ oc debug node/dev-simone-1-jzscx-worker-0-pb5fj
Temporary namespace openshift-debug-6cvdp is created for debugging node...
Starting pod/dev-simone-1-jzscx-worker-0-pb5fj-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.2.216
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-5.1# openssl s_client -showcerts -connect 172.30.212.197:443 -tls1_2 <<< 'Q' 2>&1
CONNECTED(00000003)
40ECA38C5B7F0000:error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1600:SSL alert number 70
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 138 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1689868773
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
sh-5.1# exit
exit
sh-4.4# exit
exit
Removing debug pod ...
Temporary namespace openshift-debug-6cvdp was removed.
stirabos@t14s:~$
EMS implementation landed in golang only at the end of may, see: https://go-review.googlesource.com/c/go/+/497376 and it's not realistically going to be backported to golang 1.20 (used to build CNV 4.14) not golang 1.19 (used to build CNV 4.13) so we don't have any realistic option to get this properly fixed before CNV 4.15. Closing as WONTFIX. The best we can do is suggesting to our customers to configure CNV with the modern tlsSecurity profile when in FIPS mode, this will force the server to negotiate only TLS 1.3 emitting a clear error on obsolete clients. Thew work for the release note is stracked in https://issues.redhat.com/browse/CNV-31344 |