Hide Forgot
This BZ is urgent because we need this change backported to 4.9.z before users start upgrading from 4.9 to 4.10, but it isn't a blocker.
melvinjoseph@mjoseph-mac Downloads % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False 6m5s Cluster version is 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest melvinjoseph@mjoseph-mac Downloads % mkdir test_customized_cert_no_san melvinjoseph@mjoseph-mac Downloads % cd test_customized_cert_no_san melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl genrsa -out caKey.pem 2048 Generating RSA private key, 2048 bit long modulus ............................................+++ ................................+++ e is 65537 (0x10001) melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl req -x509 -new -nodes -key caKey.pem -days 100000 -out caCert.pem -subj "/CN=network_edge_test_ca" melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl genrsa -out serverKey.pem 2048 Generating RSA private key, 2048 bit long modulus ...+++ .....................+++ e is 65537 (0x10001) melvinjoseph@mjoseph-mac test_customized_cert_no_san % cat > server_no_san.conf << EOF [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOF melvinjoseph@mjoseph-mac test_customized_cert_no_san % DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}') melvinjoseph@mjoseph-mac test_customized_cert_no_san % $DOMAIN zsh: command not found: apps.ci-ln-hqg0x12-72292.origin-ci-int-gce.dev.rhcloud.com melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl req -new -key serverKey.pem -out serverNoSAN.csr -subj "/CN=*.$DOMAIN" -config server_no_san.conf melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl x509 -req -in serverNoSAN.csr -CA caCert.pem -CAkey caKey.pem -CAcreateserial -out serverCertNoSAN.pem -days 100000 -extensions v3_req -extfile server_no_san.conf Signature ok subject=/CN=*.apps.ci-ln-hqg0x12-72292.origin-ci-int-gce.dev.rhcloud.com Getting CA Private Key melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest False False True 19m OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.ci-ln-hqg0x12-72292.origin-ci-int-gce.dev.rhcloud.com/healthz": x509: certificate relies on legacy Common Name field, use SANs instead baremetal 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 43m cloud-controller-manager 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 46m cloud-credential 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 49m cluster-autoscaler 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 44m config-operator 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 45m console 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest False True False 19m DeploymentAvailable: 0 replicas available for console deployment... csi-snapshot-controller 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 44m dns 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 44m etcd 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 42m image-registry 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 37m ingress 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 36m insights 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 38m kube-apiserver 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 40m kube-controller-manager 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 40m kube-scheduler 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 38m kube-storage-version-migrator 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 8m18s machine-api 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 40m machine-approver 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 43m machine-config 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 43m marketplace 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 44m monitoring 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 30m network 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 45m node-tuning 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 4m50s openshift-apiserver 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 9m39s openshift-controller-manager 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 41m openshift-samples 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 37m operator-lifecycle-manager 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 43m operator-lifecycle-manager-catalog 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 44m operator-lifecycle-manager-packageserver 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 38m service-ca 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 45m storage 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 40m melvinjoseph@mjoseph-mac test_customized_cert_no_san % melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc get co ingress -o json | jq .status.conditions [ { "lastTransitionTime": "2022-03-01T04:36:11Z", "message": "The \"default\" ingress controller reports Available=True.", "reason": "IngressAvailable", "status": "True", "type": "Available" }, { "lastTransitionTime": "2022-03-01T04:36:11Z", "message": "desired and current number of IngressControllers are equal", "reason": "AsExpected", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2022-03-01T04:36:11Z", "message": "The \"default\" ingress controller reports Degraded=False.", "reason": "IngressNotDegraded", "status": "False", "type": "Degraded" }, { "lastTransitionTime": "2022-03-01T04:52:58Z", "message": "Some ingresscontrollers are not upgradeable: ingresscontroller \"default\" is not upgradeable: OperandsNotUpgradeable: One or more managed resources are not upgradeable: certificate in secret openshift-ingress/custom-certs-default has legacy Common Name (CN) but has no Subject Alternative Name (SAN) for domain: *.apps.ci-ln-hqg0x12-72292.origin-ci-int-gce.dev.rhcloud.com", "reason": "IngressControllersNotUpgradeable", "status": "False", "type": "Upgradeable" } ] VERIFIED THE CLUSTER NOT UPGRADE-ABLE. melvinjoseph@mjoseph-mac Downloads % mkdir tmp_dir melvinjoseph@mjoseph-mac Downloads % cd tmp_dir melvinjoseph@mjoseph-mac tmp_dir % curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/ca.key melvinjoseph@mjoseph-mac tmp_dir % curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/ca.pem melvinjoseph@mjoseph-mac tmp_dir % curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/openssl.conf melvinjoseph@mjoseph-mac tmp_dir % sed -i.bak "s/example.com/${domain}/g" openssl.conf melvinjoseph@mjoseph-mac tmp_dir % openssl genrsa -out apps.key 2048 Generating RSA private key, 2048 bit long modulus ..............................................................+++ .........................+++ e is 65537 (0x10001) melvinjoseph@mjoseph-mac tmp_dir % openssl req -new -config openssl.conf -key apps.key -out apps.csr melvinjoseph@mjoseph-mac tmp_dir % openssl x509 -req -CA ca.pem -CAkey ca.key -CAcreateserial -extfile openssl.conf -extensions v3_req -in apps.csr -out apps.crt -days 3650 Signature ok subject=/C=US/ST=VA/L=Somewhere/O=RedHat/OU=OpenShift QE/CN=apps Getting CA Private Key melvinjoseph@mjoseph-mac tmp_dir % openssl x509 -text -noout -in apps.crt | grep "Alternative Name" -A 1 X509v3 Subject Alternative Name: DNS:*.apps.ci-ln-hqg0x12-72292.origin-ci-int-gce.dev.rhcloud.com melvinjoseph@mjoseph-mac Downloads % oc --namespace openshift-ingress create secret tls custom-certs-default-new --cert=tmp_dir/apps.crt --key=tmp_dir/apps.key secret/custom-certs-default-new created melvinjoseph@mjoseph-mac Downloads % oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \ --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default-new"}}}' ingresscontroller.operator.openshift.io/default patched melvinjoseph@mjoseph-mac Downloads % oc create configmap user-ca-bundle2 --from-file=ca-bundle.crt=tmp_dir/ca.pem -n openshift-config configmap/user-ca-bundle2 created melvinjoseph@mjoseph-mac Downloads % oc patch proxy/cluster --patch '{"spec":{"trustedCA":{"name":"user-ca-bundle2"}}}' --type=merge proxy.config.openshift.io/cluster patched melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get secret NAME TYPE DATA AGE builder-dockercfg-wxc6v kubernetes.io/dockercfg 1 128m builder-token-clzcl kubernetes.io/service-account-token 4 129m builder-token-lj768 kubernetes.io/service-account-token 4 128m custom-certs-default kubernetes.io/tls 2 107m custom-certs-default-new kubernetes.io/tls 2 71s custom-certs-default-san kubernetes.io/tls 2 78m default-dockercfg-9gxml kubernetes.io/dockercfg 1 129m default-token-rtmwm kubernetes.io/service-account-token 4 129m default-token-zg8zs kubernetes.io/service-account-token 4 131m deployer-dockercfg-k7gwj kubernetes.io/dockercfg 1 128m deployer-token-cnwn2 kubernetes.io/service-account-token 4 129m deployer-token-cqkj9 kubernetes.io/service-account-token 4 128m router-dockercfg-lnqlc kubernetes.io/dockercfg 1 129m router-metrics-certs-default kubernetes.io/tls 2 131m router-stats-default Opaque 2 131m router-token-hzsgp kubernetes.io/service-account-token 4 129m router-token-tvff5 kubernetes.io/service-account-token 4 131m melvinjoseph@mjoseph-mac Downloads % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 15m baremetal 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 146m cloud-controller-manager 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 149m cloud-credential 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 152m cluster-autoscaler 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 147m config-operator 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 148m console 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 15m csi-snapshot-controller 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 147m dns 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 147m etcd 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 145m image-registry 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 140m ingress 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 139m insights 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 141m kube-apiserver 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 144m kube-controller-manager 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 143m kube-scheduler 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 141m kube-storage-version-migrator 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 10m machine-api 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 143m machine-approver 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 146m machine-config 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 146m marketplace 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 147m monitoring 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 133m network 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 148m node-tuning 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 84m openshift-apiserver 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 112m openshift-controller-manager 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 91m openshift-samples 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 141m operator-lifecycle-manager 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 146m operator-lifecycle-manager-catalog 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 147m operator-lifecycle-manager-packageserver 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 141m service-ca 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 148m storage 4.10.0-0.ci.test-2022-03-01-041438-ci-ln-hqg0x12-latest True False False 143m melvinjoseph@mjoseph-mac Downloads % oc get co ingress -o json | jq .status.conditions [ { "lastTransitionTime": "2022-03-01T04:36:11Z", "message": "The \"default\" ingress controller reports Available=True.", "reason": "IngressAvailable", "status": "True", "type": "Available" }, { "lastTransitionTime": "2022-03-01T04:36:11Z", "message": "desired and current number of IngressControllers are equal", "reason": "AsExpected", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2022-03-01T04:36:11Z", "message": "The \"default\" ingress controller reports Degraded=False.", "reason": "IngressNotDegraded", "status": "False", "type": "Degraded" }, { "lastTransitionTime": "2022-03-01T05:22:10Z", "reason": "IngressControllersUpgradeable", "status": "True", "type": "Upgradeable" } ]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.10.6 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1026
I am setting "No Doc Update" on this BZ because the only reason for backporting the change to 4.10 was so that we could backport it to 4.9. I am adding appropriate doc text to the 4.9 BZ, bug 2060111.