Bug 2055494 - console operator should report Upgradeable False when SAN-less certs are used
Summary: console operator should report Upgradeable False when SAN-less certs are used
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Management Console
Version: 4.9
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.9.z
Assignee: Jakub Hadvig
QA Contact: Yadan Pei
URL:
Whiteboard:
Depends On: 2031839
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-17 05:42 UTC by Yadan Pei
Modified: 2022-08-09 14:01 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-09 14:00:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
changing status back-and-force (1.07 MB, video/quicktime)
2022-06-10 02:58 UTC, Yadan Pei
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift console-operator pull 638 0 None open Bug 2055494: console-operator should report Upgradeable False when SAN-less certs are used 2022-02-21 21:22:15 UTC
Github openshift console-operator pull 662 0 None open Bug 2055494: Distinguish between route conditions 2022-07-08 10:22:58 UTC
Red Hat Product Errata RHSA-2022:5879 0 None None None 2022-08-09 14:01:27 UTC

Description Yadan Pei 2022-02-17 05:42:36 UTC
Description of problem:
console operator reports Upgradeable: True when not SAN certs are used

Version-Release number of selected component (if applicable):
4.9.22

How reproducible:
Always

Steps to Reproduce:
1. Prepare a customized cert without SAN:
mkdir test_customized_oauth_cert_no_san
cd test_customized_oauth_cert_no_san
openssl genrsa -out caKey.pem 2048
openssl req -x509 -new -nodes -key caKey.pem -days 100000 -out caCert.pem -subj "/CN=xxia_test_ca"
openssl genrsa -out serverKey.pem 2048
cat > server_no_san.conf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF

CUSTOM_DOMAIN=qe1.devcluster.openshift.com
openssl req -new -key serverKey.pem -out serverNoSAN.csr -subj "/CN=*.$CUSTOM_DOMAIN" -config server_no_san.conf
openssl x509 -req -in serverNoSAN.csr -CA caCert.pem -CAkey caKey.pem -CAcreateserial -out serverCertNoSAN.pem -days 100000 -extensions v3_req -extfile server_no_san.conf

2. Make sure the customized route resolvable:
the customized route `yapei-san-less-test.qe1.devcluster.openshift.com
` need to be resolvable. We can add an A record in route53

3. Create secrets with customized cert and key
$ oc --namespace openshift-config create secret tls custom-console-component --cert=serverCertNoSAN.pem --key=serverKey.pem
secret/custom-console-component created

4. update console to use customized hostname and certs
$ oc get ingress.config cluster -o json | jq .spec
{
  "componentRoutes": [
    {
      "hostname": "yapei-san-less-test.qe1.devcluster.openshift.com",
      "name": "console",
      "namespace": "openshift-console",
      "servingCertKeyPairSecret": {
        "name": "custom-console-component"
      }
    }
  ],
  "domain": "apps.yapei49.qe.gcp.devcluster.openshift.com"
}

5. wait for new console pods created and new customized route created
$ oc get route -n openshift-console
NAME             HOST/PORT                                                                  PATH   SERVICES           PORT                    TERMINATION          WILDCARD
console          console-openshift-console.apps.yapei49.qe.gcp.devcluster.openshift.com            console-redirect   custom-route-redirect   edge/Redirect        None
console-custom   yapei-san-less-test.qe1.devcluster.openshift.com                                  console            https                   reencrypt/Redirect   None
downloads        downloads-openshift-console.apps.yapei49.qe.gcp.devcluster.openshift.com          downloads          http                    edge/Redirect        None

6. visit customized console hostname 'yapei-san-less-test.qe1.devcluster.openshift.com' and verify it is using customized certs

7. check console operator Upgradeable status
$ oc get co console -o json | jq .status.conditions
[
  {
    "lastTransitionTime": "2022-02-17T02:17:48Z",
    "message": "All is well",
    "reason": "AsExpected",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2022-02-17T03:17:00Z",
    "message": "All is well",
    "reason": "AsExpected",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2022-02-17T02:19:17Z",
    "message": "All is well",
    "reason": "AsExpected",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2022-02-17T02:17:47Z",
    "message": "All is well",
    "reason": "AsExpected",
    "status": "True",
    "type": "Upgradeable"
  }
]

Actual results:
7. console operator still reports Upgradeable: True

Expected results:
7. console operator should report Upgradeable: False when not SAN certs are used to remind user before they upgrade to 4.10, if some components using SAN-less certs after upgrading to 4.10, the cluster will be broken

Additional info:
some background info can be found from 
https://bugzilla.redhat.com/show_bug.cgi?id=2052467

Comment 2 Xingxing Xia 2022-02-24 02:28:14 UTC
Adding a kindly reminder note:
When verifying the bug, we should cover two checkpoints:
One is as said in above "Expected results".
The other is, when user updates it with a cert with SAN, it should be back to be normal to unblock user from upgrading.

Comment 6 Yadan Pei 2022-06-10 02:50:27 UTC
1. setup 4.9.38 cluster
% oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.38    True        False         51m     Cluster version is 4.9.38

2. Prepare a customized cert without SAN with steps in Description, and create secrets with customized cert and key
$ oc --namespace openshift-config create secret tls custom-console-component --cert=serverCertNoSAN.pem --key=serverKey.pem
secret/custom-console-component created

3. set console with customized route
% oc patch ingress.config cluster -p '{"spec":{"componentRoutes":[{"hostname":"yapei-san-less-test.qe1.devcluster.openshift.com", "name":"console", "namespace":"openshift-console","servingCertKeyPairSecret":{"name":"custom-console-component"}}]}}' --type merge 
ingress.config.openshift.io/cluster patched
% oc get ingress.config cluster -o json | jq .spec
{
  "componentRoutes": [
    {
      "hostname": "yapei-san-less-test.qe1.devcluster.openshift.com",
      "name": "console",
      "namespace": "openshift-console",
      "servingCertKeyPairSecret": {
        "name": "custom-console-component"
      }
    }
  ],
  "domain": "apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com"
}

4. no new console pods are created, no 'console-custom' route will be created
% oc get route -n openshift-console
NAME        HOST/PORT                                                                                PATH   SERVICES    PORT    TERMINATION          WILDCARD
console     console-openshift-console.apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com            console     https   reencrypt/Redirect   None
downloads   downloads-openshift-console.apps.ci-ln-6xw2pn2-72292.origin-ci-int-gce.dev.rhcloud.com          downloads   http    edge/Redirect        None


% oc get co | grep console
console                                    4.9.38    False       False         True       6m40s   RouteHealthAvailable: console route is not admitted


at the beginning, it reports Upgradeable: False with reason CustomRouteSync_InvalidCustomTLSSecret

% oc describe co console
Name:         console
Namespace:    
..............
Status:
  Conditions:
    Last Transition Time:  2022-06-10T02:41:25Z
    Message:               CustomRouteSyncDegraded: failed to verify custom certificate PEM: custom TLS certificate has no SAN
RouteHealthDegraded: console route is not admitted
SyncLoopRefreshDegraded: routes.route.openshift.io "console-custom" not found
    Reason:                CustomRouteSync_InvalidCustomTLSSecret::RouteHealth_RouteNotAdmitted::SyncLoopRefresh_FailedGet
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2022-06-10T01:48:09Z
    Message:               All is well
    Reason:                AsExpected
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2022-06-10T02:39:44Z
    Message:               RouteHealthAvailable: console route is not admitted
    Reason:                RouteHealth_RouteNotAdmitted
    Status:                False
    Type:                  Available
    Last Transition Time:  2022-06-10T02:41:45Z
    Message:               CustomRouteSyncUpgradeable: failed to verify custom certificate PEM: custom TLS certificate has no SAN
    Reason:                CustomRouteSync_InvalidCustomTLSSecret
    Status:                False
    Type:                  Upgradeable


then after several minutes, it reports Upgradeable: True again

% oc describe co console
Name:         console
Namespace:    
...............
Status:
  Conditions:
    Last Transition Time:  2022-06-10T02:41:25Z
    Message:               RouteHealthDegraded: console route is not admitted
SyncLoopRefreshDegraded: routes.route.openshift.io "console-custom" not found
    Reason:                RouteHealth_RouteNotAdmitted::SyncLoopRefresh_FailedGet
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2022-06-10T01:48:09Z
    Message:               All is well
    Reason:                AsExpected
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2022-06-10T02:39:44Z
    Message:               RouteHealthAvailable: console route is not admitted
    Reason:                RouteHealth_RouteNotAdmitted
    Status:                False
    Type:                  Available
    Last Transition Time:  2022-06-10T02:49:00Z
    Message:               All is well
    Reason:                AsExpected
    Status:                True
    Type:                  Upgradeable

Comment 7 Yadan Pei 2022-06-10 02:57:32 UTC
console operator keeps switching its state from Upgradeable:True to Upgradeable: False back-and-force, which leads to co/console switches status between 'Unavailable' and 'Cannot update', see screen recording

Comment 8 Yadan Pei 2022-06-10 02:58:50 UTC
Created attachment 1888521 [details]
changing status back-and-force

Comment 11 Yadan Pei 2022-07-19 02:53:38 UTC
$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.9.0-0.nightly-2022-07-18-182844   True        False         False      20m     
baremetal                                  4.9.0-0.nightly-2022-07-18-182844   True        False         False      33m     
cloud-controller-manager                   4.9.0-0.nightly-2022-07-18-182844   True        False         False      35m     
cloud-credential                           4.9.0-0.nightly-2022-07-18-182844   True        False         False      34m     
cluster-autoscaler                         4.9.0-0.nightly-2022-07-18-182844   True        False         False      32m     
config-operator                            4.9.0-0.nightly-2022-07-18-182844   True        False         False      34m     
console                                    4.9.0-0.nightly-2022-07-18-182844   False       False         True       6m46s   RouteHealthAvailable: console route is not admitted


$ oc get co console -o json | jq .status.conditions
[
  {
    "lastTransitionTime": "2022-07-19T02:42:37Z",
    "message": "ConsoleCustomRouteSyncDegraded: failed to verify custom certificate PEM: custom TLS certificate has no SAN\nRouteHealthDegraded: routes.route.openshift.io \"console-custom\" not found\nSyncLoopRefreshDegraded: routes.route.openshift.io \"console-custom\" not found",
    "reason": "ConsoleCustomRouteSync_InvalidCustomTLSSecret::RouteHealth_FailedRouteGet::SyncLoopRefresh_FailedGet",
    "status": "True",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2022-07-19T02:21:40Z",
    "message": "All is well",
    "reason": "AsExpected",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2022-07-19T02:40:39Z",
    "message": "RouteHealthAvailable: console route is not admitted",
    "reason": "RouteHealth_RouteNotAdmitted",
    "status": "False",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2022-07-19T02:40:27Z",
    "message": "ConsoleCustomRouteSyncUpgradeable: failed to verify custom certificate PEM: custom TLS certificate has no SAN",
    "reason": "ConsoleCustomRouteSync_InvalidCustomTLSSecret",
    "status": "False",
    "type": "Upgradeable"
  }
]
Upgradeable: False && Available: False && Degraded: True

Comment 12 Yadan Pei 2022-07-19 02:56:19 UTC
$ oc get route -n openshift-console
NAME        HOST/PORT                                                                                PATH   SERVICES    PORT    TERMINATION          WILDCARD
console     console-openshift-console.apps.ci-ln-zr9fy2t-72292.origin-ci-int-gce.dev.rhcloud.com            console     https   reencrypt/Redirect   None
downloads   downloads-openshift-console.apps.ci-ln-zr9fy2t-72292.origin-ci-int-gce.dev.rhcloud.com          downloads   http    edge/Redirect        None

Comment 13 Yadan Pei 2022-07-19 07:42:11 UTC
the original issue is fixed based on comment 11, moving to VERIFIED

Comment 18 errata-xmlrpc 2022-08-09 14:00:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.9.45 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5879


Note You need to log in before you can comment on or make changes to this bug.