Description of problem: HCO should pick TLSProfile from apiserver if not specified in HCO explicitly. Version-Release number of selected component (if applicable): 4.12 How reproducible: Always Steps to Reproduce: 1.Set Old profile on cluster level (oc edit apiserver cluster) 2.check HCO - it does not have Old profile inside 3. check connection to HCO - it allows tls v1.2 and 1.3 only 4. check Kubevirt - it has tls configuration updated 5. check connection to Kubevirt - it allows all versions: 1.0, 1.1, 1.2, 1.3 Actual results: HCO doesn't pick apiserver ciphers like kuebirt /SSP are picking up. Expected results: if you don't have any explicit value on HCO, all the components should comply with the cluster wide setting on apiserver Additional info: with custom profile cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get apiserver cluster -ojsonpath={.spec.tlsSecurityProfile} {"custom":{"ciphers":["DHE-RSA-AES256-GCM-SHA384","ECDHE-ECDSA-AES128-GCM-SHA256"],"minTLSVersion":"VersionTLS12"},"type":"Custom"} cnv-qcnv-qe-jenkins@cnv-qe-infra-01:~$ oc get ssp ssp-kubevirt-hyperconverged -ojsonpath={.spec.tlsSecurityProfile} {"custom":{"ciphers":["DHE-RSA-AES256-GCM-SHA384","ECDHE-ECDSA-AES128-GCM-SHA256"],"minTLSVersion":"VersionTLS12"},"type":"Custom"} cnv-qcnv-qe-jenkins@cnv-qe-infra-01:~$ oc get hco kubevirt-hyperconverged -ojsonpath={.spec.tlsSecurityProfile}
Test Env: Deployed: OCP-4.12.0-rc.0 Deployed: CNV-v4.12.0-693 Test Case 1: 1. Patch apiserver. oc patch apiserver --type=json cluster -p '[{"op": "replace", "path": /spec/tlsSecurityProfile, "value": {custom: {minTLSVersion: "VersionTLS12", ciphers: ["ECDHE-RSA-AES128-GCM-SHA256"]}, type: "Custom"} }]' 2. Make sure HCO has no explicit value [cnv-qe-jenkins@c01-gknov8-gk55b-executor ~]$ oc get hco kubevirt-hyperconverged -n openshift-cnv -ojsonpath={.spec.tlsSecurityProfile} ==> It didn't pick custom settings for tlsSecurityProfile. [cnv-qe-jenkins@c01-gknov8-gk55b-executor ~]$ 3. $ oc get ssp ssp-kubevirt-hyperconverged -n openshift-cnv -ojsonpath={.spec.tlsSecurityProfile} {"custom":{"ciphers":["ECDHE-RSA-AES128-GCM-SHA256"],"minTLSVersion":"VersionTLS12"},"type":"Custom"} Test Case 2: 1. set old in apiserver. $ oc get apiserver cluster -ojsonpath={.spec.tlsSecurityProfile} {"old":{},"type":"Old"} 2. Check if same is propogated to hco and ssp [cnv-qe-jenkins@c01-gknov8-gk55b-executor ~]$ oc get hco kubevirt-hyperconverged -n openshift-cnv -ojsonpath={.spec.tlsSecurityProfile} [cnv-qe-jenkins@c01-gknov8-gk55b-executor ~]$ oc get ssp ssp-kubevirt-hyperconverged -n openshift-cnv -ojsonpath={.spec.tlsSecurityProfile} {"old":{},"type":"Old"} Test Case 3: For Kubevirt , tlsConfiguration: ciphers: - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - TLS_RSA_WITH_AES_128_GCM_SHA256 - TLS_RSA_WITH_AES_256_GCM_SHA384 - TLS_RSA_WITH_AES_128_CBC_SHA256 - TLS_RSA_WITH_AES_128_CBC_SHA - TLS_RSA_WITH_AES_256_CBC_SHA - TLS_RSA_WITH_3DES_EDE_CBC_SHA minTLSVersion: VersionTLS10 Test Case 3: Incorrect cipher value 1. $ oc get apiserver cluster -ojsonpath={.spec.tlsSecurityProfile} {"custom":{"ciphers":["ECDHE-RSA-AES128-GCM-SHA256"],"minTLSVersion":"VersionTLS12"},"type":"Custom"} [cnv-qe-jenkins@c01-gknov8-gk55b-executor ~]$ oc get apiserver cluster -ojsonpath={.spec.tlsSecurityProfilpatch apiserver --type=json cluster -p '[{"op": "replace", "path": /spec/tlsSecurityProfile, "value": {custom: {minTLSVersion: "VersionTLS12", ciphers: ["ECDHE-RSA-AES128-GCM-SHA256","testing"]}, type: "Custom"} }]' apiserver.config.openshift.io/cluster patched [cnv-qe-jenkins@c01-gknov8-gk55b-executor ~]$ oc get apiserver cluster -ojsonpath={.spec.tlsSecurityProfile} {"custom":{"ciphers":["ECDHE-RSA-AES128-GCM-SHA256","testing"],"minTLSVersion":"VersionTLS12"},"type":"Custom"} 2. SSP $ oc get ssp ssp-kubevirt-hyperconverged -n openshift-cnv -ojsonpath={.spec.tlsSecurityProfile} {"custom":{"ciphers":["ECDHE-RSA-AES128-GCM-SHA256"],"minTLSVersion":"VersionTLS12"},"type":"Custom"} 3. hco Throws error {"level":"error","ts":1668737691.44483,"logger":"controller_hyperconverged","msg":"failed to ensure an operand","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","error":"NetworkAddonsConfig.networkaddonsoperator.network.kubevirt.io \"cluster\" is invalid: spec.tlsSecurityProfile.custom.ciphers[1]: Unsupported value: \"testing\": supported values: \"TLS_AES_128_GCM_SHA256\", \"TLS_AES_256_GCM_SHA384\", \"TLS_CHACHA20_POLY1305_SHA256\", \"ECDHE-ECDSA-AES128-GCM-SHA256\", \"ECDHE-RSA-AES128-GCM-SHA256\", \"ECDHE-ECDSA-AES256-GCM-SHA384\", \"ECDHE-RSA-AES256-GCM-SHA384\", \"ECDHE-ECDSA-CHACHA20-POLY1305\", \"ECDHE-RSA-CHACHA20-POLY1305\", \"DHE-RSA-AES128-GCM-SHA256\", \"DHE-RSA-AES256-GCM-SHA384\", \"TLS_AES_128_GCM_SHA256\", \"TLS_AES_256_GCM_SHA384\", \"TLS_CHACHA20_POLY1305_SHA256\", \"TLS_AES_128_GCM_SHA256\", \"TLS_AES_256_GCM_SHA384\", \"TLS_CHACHA20_POLY1305_SHA256\", \"ECDHE-ECDSA-AES128-GCM-SHA256\", \"ECDHE-RSA-AES128-GCM-SHA256\", \"ECDHE-ECDSA-AES256-GCM-SHA384\", \"ECDHE-RSA-AES256-GCM-SHA384\", \"ECDHE-ECDSA-CHACHA20-POLY1305\", \"ECDHE-RSA-CHACHA20-POLY1305\", \"DHE-RSA-AES128-GCM-SHA256\", \"DHE-RSA-AES256-GCM-SHA384\", \"DHE-RSA-CHACHA20-POLY1305\", \"ECDHE-ECDSA-AES128-SHA256\", \"ECDHE-RSA-AES128-SHA256\", \"ECDHE-ECDSA-AES128-SHA\", \"ECDHE-RSA-AES128-SHA\", \"ECDHE-ECDSA-AES256-SHA384\", \"ECDHE-RSA-AES256-SHA384\", \"ECDHE-ECDSA-AES256-SHA\", \"ECDHE-RSA-AES256-SHA\", \"DHE-RSA-AES128-SHA256\", \"DHE-RSA-AES256-SHA256\", \"AES128-GCM-SHA256\", \"AES256-GCM-SHA384\", \"AES128-SHA256\", \"AES256-SHA256\", \"AES128-SHA\", \"AES256-SHA\", \"DES-CBC3-SHA\"","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/controllers/operands.(*OperandHandler).Ensure\n\t/remote-source/app/controllers/operands/operandHandler.go:138\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).EnsureOperandAndComplete\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:504\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).doReconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:453\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).Reconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:317\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"} oc get hco kubevirt-hyperconverged -n openshift-cnv -ojsonpath={.spec.tlsSecurityProfile} --> empty 4. Kubevirt tlsConfiguration: ciphers: - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 minTLSVersion: VersionTLS12 Based on third case , it did try to check ciphers but hco didn't pick tlssecurityProfile while ssp/kubevirt/cdi all pick apiserver settings. Do we need to wait bit longer to achieve it.
Moving back to assigned, we will cover also: 3. hco Throws error {"level":"error","ts":1668737691.44483,"logger":"controller_hyperconverged","msg":"failed to ensure an operand","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","error":"NetworkAddonsConfig.networkaddonsoperator.network.kubevirt.io \"cluster\" is invalid: spec.tlsSecurityProfile.custom.ciphers[1]: Unsupported value: \"testing\": supported values: \"TLS_AES_128_GCM_SHA256\", \"TLS_AES_256_GCM_SHA384\", \"TLS_CHACHA20_POLY1305_SHA256\", \"ECDHE-ECDSA-AES128-GCM-SHA256\", \"ECDHE-RSA-AES128-GCM-SHA256\", \"ECDHE-ECDSA-AES256-GCM-SHA384\", \"ECDHE-RSA-AES256-GCM-SHA384\", \"ECDHE-ECDSA-CHACHA20-POLY1305\", \"ECDHE-RSA-CHACHA20-POLY1305\", \"DHE-RSA-AES128-GCM-SHA256\", \"DHE-RSA-AES256-GCM-SHA384\", \"TLS_AES_128_GCM_SHA256\", \"TLS_AES_256_GCM_SHA384\", \"TLS_CHACHA20_POLY1305_SHA256\", \"TLS_AES_128_GCM_SHA256\", \"TLS_AES_256_GCM_SHA384\", \"TLS_CHACHA20_POLY1305_SHA256\", \"ECDHE-ECDSA-AES128-GCM-SHA256\", \"ECDHE-RSA-AES128-GCM-SHA256\", \"ECDHE-ECDSA-AES256-GCM-SHA384\", \"ECDHE-RSA-AES256-GCM-SHA384\", \"ECDHE-ECDSA-CHACHA20-POLY1305\", \"ECDHE-RSA-CHACHA20-POLY1305\", \"DHE-RSA-AES128-GCM-SHA256\", \"DHE-RSA-AES256-GCM-SHA384\", \"DHE-RSA-CHACHA20-POLY1305\", \"ECDHE-ECDSA-AES128-SHA256\", \"ECDHE-RSA-AES128-SHA256\", \"ECDHE-ECDSA-AES128-SHA\", \"ECDHE-RSA-AES128-SHA\", \"ECDHE-ECDSA-AES256-SHA384\", \"ECDHE-RSA-AES256-SHA384\", \"ECDHE-ECDSA-AES256-SHA\", \"ECDHE-RSA-AES256-SHA\", \"DHE-RSA-AES128-SHA256\", \"DHE-RSA-AES256-SHA256\", \"AES128-GCM-SHA256\", \"AES256-GCM-SHA384\", \"AES128-SHA256\", \"AES256-SHA256\", \"AES128-SHA\", \"AES256-SHA\", \"DES-CBC3-SHA\"","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/controllers/operands.(*OperandHandler).Ensure\n\t/remote-source/app/controllers/operands/operandHandler.go:138\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).EnsureOperandAndComplete\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:504\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).doReconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:453\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).Reconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:317\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
one of the use case not working. Waiting for update .
Test Case : 1. Set old on apiserver. 2. changes get replcated in ssp configuration and ssp is able to connect with TLS1.3,1.2,1.1,1.0 Services : $ oc get services -n openshift-cnv| grep -e hco -e ssp hco-webhook-service ClusterIP 172.30.39.136 <none> 4343/TCP 6m19s ssp-operator-metrics ClusterIP 172.30.164.137 <none> 443/TCP 51m ssp-operator-service ClusterIP 172.30.43.175 <none> 9443/TCP 6m18s TLS connections SSP ================== sh-4.4# echo | openssl s_client -connect 172.30.43.175:9443 -brief -CAfile /tmp/ca.crt --tls1_2 Can't use SSL_get_servername CONNECTION ESTABLISHED Protocol version: TLSv1.2 Ciphersuite: ECDHE-ECDSA-AES128-GCM-SHA256 Peer certificate: O = "Red Hat, Inc.", CN = ssp-operator-service.openshift-cnv Hash used: SHA256 Signature type: ECDSA Verification: OK Supported Elliptic Curve Point Formats: uncompressed Server Temp Key: X25519, 253 bits DONE sh-4.4# echo | openssl s_client -connect 172.30.43.175:9443 -brief -CAfile /tmp/ca.crt --tls1_1 Can't use SSL_get_servername CONNECTION ESTABLISHED Protocol version: TLSv1.1 Ciphersuite: ECDHE-ECDSA-AES128-SHA Peer certificate: O = "Red Hat, Inc.", CN = ssp-operator-service.openshift-cnv Hash used: SHA1 Signature type: ECDSA Verification: OK Supported Elliptic Curve Point Formats: uncompressed Server Temp Key: X25519, 253 bits DONE sh-4.4# echo | openssl s_client -connect 172.30.43.175:9443 -brief -CAfile /tmp/ca.crt --tls1_3 Can't use SSL_get_servername CONNECTION ESTABLISHED Protocol version: TLSv1.3 Ciphersuite: TLS_AES_128_GCM_SHA256 Peer certificate: O = "Red Hat, Inc.", CN = ssp-operator-service.openshift-cnv Hash used: SHA256 Signature type: ECDSA Verification: OK Server Temp Key: X25519, 253 bits DONE sh-4.4# echo | openssl s_client -connect 172.30.43.175:9443 -brief -CAfile /tmp/ca.crt --tls1_0 s_client: Option unknown option -tls1_0 s_client: Use -help for summary. sh-4.4# echo | openssl s_client -connect 172.30.43.175:9443 -brief -CAfile /tmp/ca.crt --tls1 Can't use SSL_get_servername CONNECTION ESTABLISHED Protocol version: TLSv1 Ciphersuite: ECDHE-ECDSA-AES128-SHA Peer certificate: O = "Red Hat, Inc.", CN = ssp-operator-service.openshift-cnv Hash used: SHA1 Signature type: ECDSA Verification: OK Supported Elliptic Curve Point Formats: uncompressed Server Temp Key: X25519, 253 bits DONE 3. Changes doesn't get replicated in HCO configuration --> works as expected Connectivity check fails with TLS1.0,1.1. TLS connections HCO ==================== sh-4.4# echo | openssl s_client -connect 172.30.39.136:4343 -brief -CAfile /tmp/ca.crt --tls1_3 Can't use SSL_get_servername CONNECTION ESTABLISHED Protocol version: TLSv1.3 Ciphersuite: TLS_AES_128_GCM_SHA256 Peer certificate: O = "Red Hat, Inc.", CN = hco-webhook-service.openshift-cnv Hash used: SHA256 Signature type: ECDSA Verification: OK Server Temp Key: X25519, 253 bits DONE sh-4.4# echo | openssl s_client -connect 172.30.39.136:4343 -brief -CAfile /tmp/ca.crt --tls1_2 Can't use SSL_get_servername CONNECTION ESTABLISHED Protocol version: TLSv1.2 Ciphersuite: ECDHE-ECDSA-AES128-GCM-SHA256 Peer certificate: O = "Red Hat, Inc.", CN = hco-webhook-service.openshift-cnv Hash used: SHA256 Signature type: ECDSA Verification: OK Supported Elliptic Curve Point Formats: uncompressed Server Temp Key: X25519, 253 bits DONE sh-4.4# echo | openssl s_client -connect 172.30.39.136:4343 -brief -CAfile /tmp/ca.crt --tls1_1 140230858667840:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1544:SSL alert number 70 sh-4.4# echo | openssl s_client -connect 172.30.39.136:4343 -brief -CAfile /tmp/ca.crt --tls1 140629337196352:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1544:SSL alert number 70 sh-4.4# Removing debug pod ... Workaround: ========= Killing hco webhook pods result in HCO getting restarted with correct tlssecurityprofile. This needs some work to understand why we have to kill pods and then HCO is picking right configuration. Moving it to development for some review/fixes.
https://github.com/kubevirt/hyperconverged-cluster-operator/pull/2164 should address also the case in comment #4
Test Env: Deployed: OCP-4.12.0-rc.6 Deployed: CNV-v4.12.0-769 Test Setup: Non Fips cluster HCO : sh-4.4# echo | openssl s_client -connect 172.30.124.195:4343 -brief --tls1 Can't use SSL_get_servername depth=0 O = "Red Hat, Inc.", CN = hco-webhook-service.openshift-cnv verify error:num=20:unable to get local issuer certificate depth=0 O = "Red Hat, Inc.", CN = hco-webhook-service.openshift-cnv verify error:num=21:unable to verify the first certificate CONNECTION ESTABLISHED Protocol version: TLSv1 Ciphersuite: ECDHE-ECDSA-AES128-SHA Peer certificate: O = "Red Hat, Inc.", CN = hco-webhook-service.openshift-cnv Hash used: SHA1 Signature type: ECDSA Verification error: unable to verify the first certificate Supported Elliptic Curve Point Formats: uncompressed Server Temp Key: X25519, 253 bits DONE sh-4.4# echo | openssl s_client -connect 172.30.124.195:4343 -brief --tls1_1 Can't use SSL_get_servername depth=0 O = "Red Hat, Inc.", CN = hco-webhook-service.openshift-cnv verify error:num=20:unable to get local issuer certificate depth=0 O = "Red Hat, Inc.", CN = hco-webhook-service.openshift-cnv verify error:num=21:unable to verify the first certificate CONNECTION ESTABLISHED Protocol version: TLSv1.1 Ciphersuite: ECDHE-ECDSA-AES128-SHA Peer certificate: O = "Red Hat, Inc.", CN = hco-webhook-service.openshift-cnv Hash used: SHA1 Signature type: ECDSA Verification error: unable to verify the first certificate Supported Elliptic Curve Point Formats: uncompressed Server Temp Key: X25519, 253 bits DONE
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Virtualization 4.12.0 Images security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:0408