Description of problem: After setting Modern tls profile in HCO all virt system pods become in non-ready state: > virt-api-847b79c85b-gmd6p 0/1 Running 0 37m > virt-api-847b79c85b-v6fhd 0/1 Running 0 37m > virt-controller-64f9448b66-pwvcz 0/1 Running 6 (24s ago) 9h > virt-controller-64f9448b66-sw48x 0/1 Running 6 (25s ago) 9h > virt-exportproxy-744f8cc64b-7czbl 0/1 Running 0 9h > virt-exportproxy-744f8cc64b-tvf65 0/1 Running 0 9h > virt-handler-2flcz 0/1 Running 6 (35s ago) 4d15h > virt-handler-7ks9d 0/1 Running 7 (11s ago) 4d16h > virt-handler-jcfr2 0/1 Running 6 (39s ago) 4d16h > virt-operator-747b5f58f8-7gv78 0/1 Running 0 9h > virt-operator-747b5f58f8-r2929 0/1 Running 0 9 There are many error messages in pod logs: > {"component":"virt-api","level":"info","msg":"http: TLS handshake error from 10.129.2.2:57388: tls: client offered only unsupported versions: [303]\n","pos":"server.go:3197","timestamp":"2022-11-01T22:02:14.590402Z"} I see the similar errors in other CNV components (SSP, CDI and HCO itself). And it can be tricky to revert TLS cofiguration back because webhooks is also not available: > $ oc edit hco > error: hyperconvergeds.hco.kubevirt.io "kubevirt-hyperconverged" could not be patched: Internal error occurred: failed calling webhook "validate-hco.kubevirt.io": failed to call webhook: Post "https://hco-webhook-service.openshift-cnv.svc:4343/validate-hco-kubevirt-io-v1beta1-hyperconverged?timeout=10s": remote error: tls: protocol version not supported Version-Release number of selected component (if applicable): 4.12 How reproducible: Steps to Reproduce: 1. edit HCO and set tlsSecurityProfile to 'Modern' 2. check pods status and logs Actual results: all CNV components have 'TLS handshake error' errors Expected results: CNV components active and accessible with Modern profile Additional info: After some investigation it looks like the issue happens only on a cluster with FIPS mode enabled.
Denys, Can you clarify which ciphers are mandated by the "modern" profile? Is that the entire log message? It sounds like the problem might be that we simply don't have any ciphers available.
I think this bug may be related to bz 2139222, for some reason on FIPS enabled cluster only TLS v1.2 works. All other versions are blocked. Here the output for FIPS Enabled cluster with default TLS configuration (Intermediate profile) $ oc get kubevirt kubevirt-kubevirt-hyperconverged -n openshift-cnv -o json | jq .spec.configuration.tlsConfiguration { "ciphers": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256" ], "minTLSVersion": "VersionTLS12" } $ nmap --script ssl-enum-ciphers -p 1443 127.0.0.1 > PORT STATE SERVICE > 1443/tcp open ies-lm > | ssl-enum-ciphers: > | TLSv1.2: > | ciphers: > | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A > | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A > | compressors: > | NULL > | cipher preference: server > |_ least strength: A And here FIPS Disabled cluster with the default TLS profile: > $ nmap --script ssl-enum-ciphers -p 1443 127.0.0.1 > PORT STATE SERVICE > 1443/tcp open ies-lm > | ssl-enum-ciphers: > | TLSv1.2: > | ciphers: > | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A > | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A > | TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (secp256r1) - A > | compressors: > | NULL > | cipher preference: client > | TLSv1.3: > | ciphers: > | TLS_AKE_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A > | TLS_AKE_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A > | TLS_AKE_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A > | cipher preference: server > |_ least strength: A You can see the difference - cluster with FIPS mode enabled allows TLSv1.2 only, so when I set Modern profile (or minTLSVersion 1.3) - virt pods shows tls handshake errors
Moving this bug to 4.14 based on priority and capacity.
Moving this bug to 4.15 based on priority and capacity.