Description of problem: console operator reports Upgradeable: True when not SAN certs are used Version-Release number of selected component (if applicable): 4.9.22 How reproducible: Always Steps to Reproduce: 1. Prepare a customized cert without SAN: mkdir test_customized_oauth_cert_no_san cd test_customized_oauth_cert_no_san openssl genrsa -out caKey.pem 2048 openssl req -x509 -new -nodes -key caKey.pem -days 100000 -out caCert.pem -subj "/CN=xxia_test_ca" openssl genrsa -out serverKey.pem 2048 cat > server_no_san.conf << EOF [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOF CUSTOM_DOMAIN=qe1.devcluster.openshift.com openssl req -new -key serverKey.pem -out serverNoSAN.csr -subj "/CN=*.$CUSTOM_DOMAIN" -config server_no_san.conf openssl x509 -req -in serverNoSAN.csr -CA caCert.pem -CAkey caKey.pem -CAcreateserial -out serverCertNoSAN.pem -days 100000 -extensions v3_req -extfile server_no_san.conf 2. Make sure the customized route resolvable: the customized route `yapei-san-less-test.qe1.devcluster.openshift.com ` need to be resolvable. We can add an A record in route53 3. Create secrets with customized cert and key $ oc --namespace openshift-config create secret tls custom-console-component --cert=serverCertNoSAN.pem --key=serverKey.pem secret/custom-console-component created 4. update console to use customized hostname and certs $ oc get ingress.config cluster -o json | jq .spec { "componentRoutes": [ { "hostname": "yapei-san-less-test.qe1.devcluster.openshift.com", "name": "console", "namespace": "openshift-console", "servingCertKeyPairSecret": { "name": "custom-console-component" } } ], "domain": "apps.yapei49.qe.gcp.devcluster.openshift.com" } 5. wait for new console pods created and new customized route created $ oc get route -n openshift-console NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-openshift-console.apps.yapei49.qe.gcp.devcluster.openshift.com console-redirect custom-route-redirect edge/Redirect None console-custom yapei-san-less-test.qe1.devcluster.openshift.com console https reencrypt/Redirect None downloads downloads-openshift-console.apps.yapei49.qe.gcp.devcluster.openshift.com downloads http edge/Redirect None 6. visit customized console hostname 'yapei-san-less-test.qe1.devcluster.openshift.com' and verify it is using customized certs 7. check console operator Upgradeable status $ oc get co console -o json | jq .status.conditions [ { "lastTransitionTime": "2022-02-17T02:17:48Z", "message": "All is well", "reason": "AsExpected", "status": "False", "type": "Degraded" }, { "lastTransitionTime": "2022-02-17T03:17:00Z", "message": "All is well", "reason": "AsExpected", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2022-02-17T02:19:17Z", "message": "All is well", "reason": "AsExpected", "status": "True", "type": "Available" }, { "lastTransitionTime": "2022-02-17T02:17:47Z", "message": "All is well", "reason": "AsExpected", "status": "True", "type": "Upgradeable" } ] Actual results: 7. console operator still reports Upgradeable: True Expected results: 7. console operator should report Upgradeable: False when not SAN certs are used to remind user before they upgrade to 4.10, if some components using SAN-less certs after upgrading to 4.10, the cluster will be broken Additional info: some background info can be found from https://bugzilla.redhat.com/show_bug.cgi?id=2052467
Adding a kindly reminder note: When verifying the bug, we should cover two checkpoints: One is as said in above "Expected results". The other is, when user updates it with a cert with SAN, it should be back to be normal to unblock user from upgrading.
1. setup 4.9.38 cluster % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.38 True False 51m Cluster version is 4.9.38 2. Prepare a customized cert without SAN with steps in Description, and create secrets with customized cert and key $ oc --namespace openshift-config create secret tls custom-console-component --cert=serverCertNoSAN.pem --key=serverKey.pem secret/custom-console-component created 3. set console with customized route % oc patch ingress.config cluster -p '{"spec":{"componentRoutes":[{"hostname":"yapei-san-less-test.qe1.devcluster.openshift.com", "name":"console", "namespace":"openshift-console","servingCertKeyPairSecret":{"name":"custom-console-component"}}]}}' --type merge ingress.config.openshift.io/cluster patched % oc get ingress.config cluster -o json | jq .spec { "componentRoutes": [ { "hostname": "yapei-san-less-test.qe1.devcluster.openshift.com", "name": "console", "namespace": "openshift-console", "servingCertKeyPairSecret": { "name": "custom-console-component" } } ], "domain": "apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com" } 4. no new console pods are created, no 'console-custom' route will be created % oc get route -n openshift-console NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-openshift-console.apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None downloads downloads-openshift-console.apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None % oc get co | grep console console 4.9.38 False False True 6m40s RouteHealthAvailable: console route is not admitted at the beginning, it reports Upgradeable: False with reason CustomRouteSync_InvalidCustomTLSSecret % oc describe co console Name: console Namespace: .............. Status: Conditions: Last Transition Time: 2022-06-10T02:41:25Z Message: CustomRouteSyncDegraded: failed to verify custom certificate PEM: custom TLS certificate has no SAN RouteHealthDegraded: console route is not admitted SyncLoopRefreshDegraded: routes.route.openshift.io "console-custom" not found Reason: CustomRouteSync_InvalidCustomTLSSecret::RouteHealth_RouteNotAdmitted::SyncLoopRefresh_FailedGet Status: True Type: Degraded Last Transition Time: 2022-06-10T01:48:09Z Message: All is well Reason: AsExpected Status: False Type: Progressing Last Transition Time: 2022-06-10T02:39:44Z Message: RouteHealthAvailable: console route is not admitted Reason: RouteHealth_RouteNotAdmitted Status: False Type: Available Last Transition Time: 2022-06-10T02:41:45Z Message: CustomRouteSyncUpgradeable: failed to verify custom certificate PEM: custom TLS certificate has no SAN Reason: CustomRouteSync_InvalidCustomTLSSecret Status: False Type: Upgradeable then after several minutes, it reports Upgradeable: True again % oc describe co console Name: console Namespace: ............... Status: Conditions: Last Transition Time: 2022-06-10T02:41:25Z Message: RouteHealthDegraded: console route is not admitted SyncLoopRefreshDegraded: routes.route.openshift.io "console-custom" not found Reason: RouteHealth_RouteNotAdmitted::SyncLoopRefresh_FailedGet Status: True Type: Degraded Last Transition Time: 2022-06-10T01:48:09Z Message: All is well Reason: AsExpected Status: False Type: Progressing Last Transition Time: 2022-06-10T02:39:44Z Message: RouteHealthAvailable: console route is not admitted Reason: RouteHealth_RouteNotAdmitted Status: False Type: Available Last Transition Time: 2022-06-10T02:49:00Z Message: All is well Reason: AsExpected Status: True Type: Upgradeable
console operator keeps switching its state from Upgradeable:True to Upgradeable: False back-and-force, which leads to co/console switches status between 'Unavailable' and 'Cannot update', see screen recording
Created attachment 1888521 [details] changing status back-and-force
$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.9.0-0.nightly-2022-07-18-182844 True False False 20m baremetal 4.9.0-0.nightly-2022-07-18-182844 True False False 33m cloud-controller-manager 4.9.0-0.nightly-2022-07-18-182844 True False False 35m cloud-credential 4.9.0-0.nightly-2022-07-18-182844 True False False 34m cluster-autoscaler 4.9.0-0.nightly-2022-07-18-182844 True False False 32m config-operator 4.9.0-0.nightly-2022-07-18-182844 True False False 34m console 4.9.0-0.nightly-2022-07-18-182844 False False True 6m46s RouteHealthAvailable: console route is not admitted $ oc get co console -o json | jq .status.conditions [ { "lastTransitionTime": "2022-07-19T02:42:37Z", "message": "ConsoleCustomRouteSyncDegraded: failed to verify custom certificate PEM: custom TLS certificate has no SAN\nRouteHealthDegraded: routes.route.openshift.io \"console-custom\" not found\nSyncLoopRefreshDegraded: routes.route.openshift.io \"console-custom\" not found", "reason": "ConsoleCustomRouteSync_InvalidCustomTLSSecret::RouteHealth_FailedRouteGet::SyncLoopRefresh_FailedGet", "status": "True", "type": "Degraded" }, { "lastTransitionTime": "2022-07-19T02:21:40Z", "message": "All is well", "reason": "AsExpected", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2022-07-19T02:40:39Z", "message": "RouteHealthAvailable: console route is not admitted", "reason": "RouteHealth_RouteNotAdmitted", "status": "False", "type": "Available" }, { "lastTransitionTime": "2022-07-19T02:40:27Z", "message": "ConsoleCustomRouteSyncUpgradeable: failed to verify custom certificate PEM: custom TLS certificate has no SAN", "reason": "ConsoleCustomRouteSync_InvalidCustomTLSSecret", "status": "False", "type": "Upgradeable" } ] Upgradeable: False && Available: False && Degraded: True
$ oc get route -n openshift-console NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-openshift-console.apps.ci-ln-zr9fy2t-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None downloads downloads-openshift-console.apps.ci-ln-zr9fy2t-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None
the original issue is fixed based on comment 11, moving to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.9.45 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5879