Bug 2055494
| Summary: | console operator should report Upgradeable False when SAN-less certs are used | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Yadan Pei <yapei> | ||||
| Component: | Management Console | Assignee: | Jakub Hadvig <jhadvig> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Yadan Pei <yapei> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.9 | CC: | aos-bugs, wking, xxia, yapei | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.9.z | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-08-09 14:00:58 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 2031839 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
Adding a kindly reminder note: When verifying the bug, we should cover two checkpoints: One is as said in above "Expected results". The other is, when user updates it with a cert with SAN, it should be back to be normal to unblock user from upgrading. 1. setup 4.9.38 cluster
% oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.9.38 True False 51m Cluster version is 4.9.38
2. Prepare a customized cert without SAN with steps in Description, and create secrets with customized cert and key
$ oc --namespace openshift-config create secret tls custom-console-component --cert=serverCertNoSAN.pem --key=serverKey.pem
secret/custom-console-component created
3. set console with customized route
% oc patch ingress.config cluster -p '{"spec":{"componentRoutes":[{"hostname":"yapei-san-less-test.qe1.devcluster.openshift.com", "name":"console", "namespace":"openshift-console","servingCertKeyPairSecret":{"name":"custom-console-component"}}]}}' --type merge
ingress.config.openshift.io/cluster patched
% oc get ingress.config cluster -o json | jq .spec
{
"componentRoutes": [
{
"hostname": "yapei-san-less-test.qe1.devcluster.openshift.com",
"name": "console",
"namespace": "openshift-console",
"servingCertKeyPairSecret": {
"name": "custom-console-component"
}
}
],
"domain": "apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com"
}
4. no new console pods are created, no 'console-custom' route will be created
% oc get route -n openshift-console
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
console console-openshift-console.apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None
downloads downloads-openshift-console.apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None
% oc get co | grep console
console 4.9.38 False False True 6m40s RouteHealthAvailable: console route is not admitted
at the beginning, it reports Upgradeable: False with reason CustomRouteSync_InvalidCustomTLSSecret
% oc describe co console
Name: console
Namespace:
..............
Status:
Conditions:
Last Transition Time: 2022-06-10T02:41:25Z
Message: CustomRouteSyncDegraded: failed to verify custom certificate PEM: custom TLS certificate has no SAN
RouteHealthDegraded: console route is not admitted
SyncLoopRefreshDegraded: routes.route.openshift.io "console-custom" not found
Reason: CustomRouteSync_InvalidCustomTLSSecret::RouteHealth_RouteNotAdmitted::SyncLoopRefresh_FailedGet
Status: True
Type: Degraded
Last Transition Time: 2022-06-10T01:48:09Z
Message: All is well
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: 2022-06-10T02:39:44Z
Message: RouteHealthAvailable: console route is not admitted
Reason: RouteHealth_RouteNotAdmitted
Status: False
Type: Available
Last Transition Time: 2022-06-10T02:41:45Z
Message: CustomRouteSyncUpgradeable: failed to verify custom certificate PEM: custom TLS certificate has no SAN
Reason: CustomRouteSync_InvalidCustomTLSSecret
Status: False
Type: Upgradeable
then after several minutes, it reports Upgradeable: True again
% oc describe co console
Name: console
Namespace:
...............
Status:
Conditions:
Last Transition Time: 2022-06-10T02:41:25Z
Message: RouteHealthDegraded: console route is not admitted
SyncLoopRefreshDegraded: routes.route.openshift.io "console-custom" not found
Reason: RouteHealth_RouteNotAdmitted::SyncLoopRefresh_FailedGet
Status: True
Type: Degraded
Last Transition Time: 2022-06-10T01:48:09Z
Message: All is well
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: 2022-06-10T02:39:44Z
Message: RouteHealthAvailable: console route is not admitted
Reason: RouteHealth_RouteNotAdmitted
Status: False
Type: Available
Last Transition Time: 2022-06-10T02:49:00Z
Message: All is well
Reason: AsExpected
Status: True
Type: Upgradeable
console operator keeps switching its state from Upgradeable:True to Upgradeable: False back-and-force, which leads to co/console switches status between 'Unavailable' and 'Cannot update', see screen recording Created attachment 1888521 [details]
changing status back-and-force
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.9.0-0.nightly-2022-07-18-182844 True False False 20m
baremetal 4.9.0-0.nightly-2022-07-18-182844 True False False 33m
cloud-controller-manager 4.9.0-0.nightly-2022-07-18-182844 True False False 35m
cloud-credential 4.9.0-0.nightly-2022-07-18-182844 True False False 34m
cluster-autoscaler 4.9.0-0.nightly-2022-07-18-182844 True False False 32m
config-operator 4.9.0-0.nightly-2022-07-18-182844 True False False 34m
console 4.9.0-0.nightly-2022-07-18-182844 False False True 6m46s RouteHealthAvailable: console route is not admitted
$ oc get co console -o json | jq .status.conditions
[
{
"lastTransitionTime": "2022-07-19T02:42:37Z",
"message": "ConsoleCustomRouteSyncDegraded: failed to verify custom certificate PEM: custom TLS certificate has no SAN\nRouteHealthDegraded: routes.route.openshift.io \"console-custom\" not found\nSyncLoopRefreshDegraded: routes.route.openshift.io \"console-custom\" not found",
"reason": "ConsoleCustomRouteSync_InvalidCustomTLSSecret::RouteHealth_FailedRouteGet::SyncLoopRefresh_FailedGet",
"status": "True",
"type": "Degraded"
},
{
"lastTransitionTime": "2022-07-19T02:21:40Z",
"message": "All is well",
"reason": "AsExpected",
"status": "False",
"type": "Progressing"
},
{
"lastTransitionTime": "2022-07-19T02:40:39Z",
"message": "RouteHealthAvailable: console route is not admitted",
"reason": "RouteHealth_RouteNotAdmitted",
"status": "False",
"type": "Available"
},
{
"lastTransitionTime": "2022-07-19T02:40:27Z",
"message": "ConsoleCustomRouteSyncUpgradeable: failed to verify custom certificate PEM: custom TLS certificate has no SAN",
"reason": "ConsoleCustomRouteSync_InvalidCustomTLSSecret",
"status": "False",
"type": "Upgradeable"
}
]
Upgradeable: False && Available: False && Degraded: True
$ oc get route -n openshift-console NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-openshift-console.apps.ci-ln-zr9fy2t-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None downloads downloads-openshift-console.apps.ci-ln-zr9fy2t-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None the original issue is fixed based on comment 11, moving to VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.9.45 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5879 |
Description of problem: console operator reports Upgradeable: True when not SAN certs are used Version-Release number of selected component (if applicable): 4.9.22 How reproducible: Always Steps to Reproduce: 1. Prepare a customized cert without SAN: mkdir test_customized_oauth_cert_no_san cd test_customized_oauth_cert_no_san openssl genrsa -out caKey.pem 2048 openssl req -x509 -new -nodes -key caKey.pem -days 100000 -out caCert.pem -subj "/CN=xxia_test_ca" openssl genrsa -out serverKey.pem 2048 cat > server_no_san.conf << EOF [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOF CUSTOM_DOMAIN=qe1.devcluster.openshift.com openssl req -new -key serverKey.pem -out serverNoSAN.csr -subj "/CN=*.$CUSTOM_DOMAIN" -config server_no_san.conf openssl x509 -req -in serverNoSAN.csr -CA caCert.pem -CAkey caKey.pem -CAcreateserial -out serverCertNoSAN.pem -days 100000 -extensions v3_req -extfile server_no_san.conf 2. Make sure the customized route resolvable: the customized route `yapei-san-less-test.qe1.devcluster.openshift.com ` need to be resolvable. We can add an A record in route53 3. Create secrets with customized cert and key $ oc --namespace openshift-config create secret tls custom-console-component --cert=serverCertNoSAN.pem --key=serverKey.pem secret/custom-console-component created 4. update console to use customized hostname and certs $ oc get ingress.config cluster -o json | jq .spec { "componentRoutes": [ { "hostname": "yapei-san-less-test.qe1.devcluster.openshift.com", "name": "console", "namespace": "openshift-console", "servingCertKeyPairSecret": { "name": "custom-console-component" } } ], "domain": "apps.yapei49.qe.gcp.devcluster.openshift.com" } 5. wait for new console pods created and new customized route created $ oc get route -n openshift-console NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-openshift-console.apps.yapei49.qe.gcp.devcluster.openshift.com console-redirect custom-route-redirect edge/Redirect None console-custom yapei-san-less-test.qe1.devcluster.openshift.com console https reencrypt/Redirect None downloads downloads-openshift-console.apps.yapei49.qe.gcp.devcluster.openshift.com downloads http edge/Redirect None 6. visit customized console hostname 'yapei-san-less-test.qe1.devcluster.openshift.com' and verify it is using customized certs 7. check console operator Upgradeable status $ oc get co console -o json | jq .status.conditions [ { "lastTransitionTime": "2022-02-17T02:17:48Z", "message": "All is well", "reason": "AsExpected", "status": "False", "type": "Degraded" }, { "lastTransitionTime": "2022-02-17T03:17:00Z", "message": "All is well", "reason": "AsExpected", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2022-02-17T02:19:17Z", "message": "All is well", "reason": "AsExpected", "status": "True", "type": "Available" }, { "lastTransitionTime": "2022-02-17T02:17:47Z", "message": "All is well", "reason": "AsExpected", "status": "True", "type": "Upgradeable" } ] Actual results: 7. console operator still reports Upgradeable: True Expected results: 7. console operator should report Upgradeable: False when not SAN certs are used to remind user before they upgrade to 4.10, if some components using SAN-less certs after upgrading to 4.10, the cluster will be broken Additional info: some background info can be found from https://bugzilla.redhat.com/show_bug.cgi?id=2052467