Bug 2060111
| Summary: | ingress operator should report Upgradeable False to remind user before upgrade to 4.10 when Non-SAN certs are used | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
| Component: | Networking | Assignee: | Miciah Dashiel Butler Masters <mmasters> |
| Networking sub component: | router | QA Contact: | Melvin Joseph <mjoseph> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | urgent | ||
| Priority: | high | CC: | aos-bugs, hongli, mfisher, mmasters, wking, xxia |
| Version: | 4.9 | Keywords: | Upgrades |
| Target Milestone: | --- | ||
| Target Release: | 4.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: Users could configure an IngressController with a default certificate with a Common Name (CN) for the IngressController's domain but no Subject Alternative Name (SAN) for the same.
Consequence: Clients built using Go 1.17 reject certificates without SANs. OpenShift 4.10 is built using Go 1.17, which means that various operators that connect to Routes that use the default certificate would reject the certificate and fail to complete the TLS handshake after upgrading to OpenShift 4.10.
Fix: A check was added to the ingress operator, which will set Upgradeable=False on an IngressController if it has has a problematic default certificate. This status condition will propagate to the ingress ClusterOperator to block upgrading to 4.10.
Result: OpenShift 4.9 blocks upgrades to 4.10 if an IngressController has a problematic default certificate.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-29 07:16:22 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2059210 | ||
| Bug Blocks: | |||
|
Comment 1
Miciah Dashiel Butler Masters
2022-03-02 17:38:46 UTC
W. Trevor King pointed <https://github.com/openshift/cluster-ingress-operator/pull/711> out to me, which should prevent the issue with born-as-4.1 clusters by setting status.platformStatus in the cluster infrastructure config object on upgraded clusters. I am therefore setting blocker- on this BZ. melvinjoseph@mjoseph-mac Downloads % oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False 106m Cluster version is 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest
melvinjoseph@mjoseph-mac Downloads % mkdir test_customized_cert_no_san
melvinjoseph@mjoseph-mac Downloads % cd test_customized_cert_no_san
melvinjoseph@mjoseph-mac test_customized_cert_no_san % ls
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl genrsa -out caKey.pem 2048
Generating RSA private key, 2048 bit long modulus
...........................+++
............................+++
e is 65537 (0x10001)
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl req -x509 -new -nodes -key caKey.pem -days 100000 -out caCert.pem -subj "/CN=network_edge_test_ca"
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl genrsa -out serverKey.pem 2048
Generating RSA private key, 2048 bit long modulus
....................+++
............................................................................................................................+++
e is 65537 (0x10001)
melvinjoseph@mjoseph-mac test_customized_cert_no_san %
melvinjoseph@mjoseph-mac test_customized_cert_no_san % cat > server_no_san.conf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF
melvinjoseph@mjoseph-mac test_customized_cert_no_san % DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')
melvinjoseph@mjoseph-mac test_customized_cert_no_san % $DOMAIN
zsh: command not found: apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl req -new -key serverKey.pem -out serverNoSAN.csr -subj "/CN=*.$DOMAIN" -config server_no_san.conf
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl x509 -req -in serverNoSAN.csr -CA caCert.pem -CAkey caKey.pem -CAcreateserial -out serverCertNoSAN.pem -days 100000 -extensions v3_req -extfile server_no_san.conf
Signature ok
subject=/CN=*.apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com
Getting CA Private Key
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc --namespace openshift-ingress create secret tls custom-certs-default --cert=serverCertNoSAN.pem --key=serverKey.pem
secret/custom-certs-default created
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
--patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'
ingresscontroller.operator.openshift.io/default patched
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc create configmap user-ca-bundle --from-file=ca-bundle.crt=caCert.pem -n openshift-config
configmap/user-ca-bundle created
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc patch proxy/cluster --patch '{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}' --type=merge
proxy.config.openshift.io/cluster patched
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 18m
baremetal 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 145m
cloud-controller-manager 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 148m
cloud-credential 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 151m
cluster-autoscaler 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 145m
config-operator 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 146m
console 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 19m
csi-snapshot-controller 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 146m
dns 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 144m
etcd 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 144m
image-registry 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 139m
ingress 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 138m
insights 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 140m
kube-apiserver 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 143m
kube-controller-manager 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 144m
kube-scheduler 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 144m
kube-storage-version-migrator 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 146m
machine-api 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 141m
machine-approver 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 144m
machine-config 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 143m
marketplace 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 145m
monitoring 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 137m
network 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 147m
node-tuning 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 118s
openshift-apiserver 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 140m
openshift-controller-manager 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 145m
openshift-samples 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 140m
operator-lifecycle-manager 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 146m
operator-lifecycle-manager-catalog 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 146m
operator-lifecycle-manager-packageserver 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 141m
service-ca 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 146m
storage 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 145m
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc get co ingress -o json | jq .status.conditions
[
{
"lastTransitionTime": "2022-03-03T03:11:57Z",
"message": "The \"default\" ingress controller reports Available=True.",
"reason": "IngressAvailable",
"status": "True",
"type": "Available"
},
{
"lastTransitionTime": "2022-03-03T03:11:57Z",
"message": "desired and current number of IngressControllers are equal",
"reason": "AsExpected",
"status": "False",
"type": "Progressing"
},
{
"lastTransitionTime": "2022-03-03T03:11:57Z",
"message": "The \"default\" ingress controller reports Degraded=False.",
"reason": "IngressNotDegraded",
"status": "False",
"type": "Degraded"
},
{
"lastTransitionTime": "2022-03-03T05:10:55Z",
"message": "Some ingresscontrollers are not upgradeable: ingresscontroller \"default\" is not upgradeable: OperandsNotUpgradeable: One or more managed resources are not upgradeable: certificate in secret openshift-ingress/custom-certs-default has legacy Common Name (CN) but has no Subject Alternative Name (SAN) for domain: *.apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com",
"reason": "IngressControllersNotUpgradeable",
"status": "False",
"type": "Upgradeable"
}
]
FIRST PART IS VERIFIED.
melvinjoseph@mjoseph-mac Downloads % mkdir tmp_dir
melvinjoseph@mjoseph-mac Downloads % cd tmp_dir
melvinjoseph@mjoseph-mac tmp_dir % curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/ca.key
melvinjoseph@mjoseph-mac tmp_dir % curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/ca.pem
melvinjoseph@mjoseph-mac tmp_dir % curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/openssl.conf
melvinjoseph@mjoseph-mac tmp_dir % $DOMAIN
zsh: command not found: apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com
melvinjoseph@mjoseph-mac tmp_dir % sed -i.bak "s/example.com/${DOMAIN}/g" openssl.conf
melvinjoseph@mjoseph-mac tmp_dir % LS
ca.key ca.pem openssl.conf openssl.conf.bak
melvinjoseph@mjoseph-mac tmp_dir % openssl genrsa -out apps.key 2048
Generating RSA private key, 2048 bit long modulus
....................................+++
.............................+++
e is 65537 (0x10001)
melvinjoseph@mjoseph-mac tmp_dir % openssl req -new -config openssl.conf -key apps.key -out apps.csr
melvinjoseph@mjoseph-mac tmp_dir % openssl x509 -req -CA ca.pem -CAkey ca.key -CAcreateserial -extfile openssl.conf -extensions v3_req -in apps.csr -out apps.crt -days 3650
Signature ok
subject=/C=US/ST=VA/L=Somewhere/O=RedHat/OU=OpenShift QE/CN=apps
Getting CA Private Key
melvinjoseph@mjoseph-mac tmp_dir % openssl x509 -text -noout -in apps.crt | grep "Alternative Name" -A 1
X509v3 Subject Alternative Name:
DNS:*.apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com
melvinjoseph@mjoseph-mac tmp_dir % oc --namespace openshift-ingress create secret tls custom-certs-default-new --cert=apps.crt --key=apps.key
secret/custom-certs-default-new created
melvinjoseph@mjoseph-mac tmp_dir % oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
--patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default-new"}}}'
ingresscontroller.operator.openshift.io/default patched
melvinjoseph@mjoseph-mac tmp_dir % oc create configmap user-ca-bundle2 --from-file=ca-bundle.crt=ca.pem -n openshift-config
configmap/user-ca-bundle2 created
melvinjoseph@mjoseph-mac tmp_dir % oc patch proxy/cluster --patch '{"spec":{"trustedCA":{"name":"user-ca-bundle2"}}}' --type=merge
proxy.config.openshift.io/cluster patched
melvinjoseph@mjoseph-mac tmp_dir % oc -n openshift-ingress get secret
NAME TYPE DATA AGE
builder-dockercfg-szphp kubernetes.io/dockercfg 1 148m
builder-token-dpp78 kubernetes.io/service-account-token 4 148m
builder-token-hnk59 kubernetes.io/service-account-token 4 148m
custom-certs-default kubernetes.io/tls 2 26m
custom-certs-default-new kubernetes.io/tls 2 71s
default-dockercfg-4p2wd kubernetes.io/dockercfg 1 148m
default-token-96crm kubernetes.io/service-account-token 4 148m
default-token-s6s2x kubernetes.io/service-account-token 4 148m
deployer-dockercfg-g2p5b kubernetes.io/dockercfg 1 148m
deployer-token-9np2n kubernetes.io/service-account-token 4 148m
deployer-token-x44wx kubernetes.io/service-account-token 4 148m
router-dockercfg-qdb9f kubernetes.io/dockercfg 1 148m
router-metrics-certs-default kubernetes.io/tls 2 148m
router-stats-default Opaque 2 148m
router-token-9nsp5 kubernetes.io/service-account-token 4 148m
router-token-pps4x kubernetes.io/service-account-token 4 148m
melvinjoseph@mjoseph-mac tmp_dir % oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 47m
baremetal 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 174m
cloud-controller-manager 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 177m
cloud-credential 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 3h
cluster-autoscaler 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 174m
config-operator 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 176m
console 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 22m
csi-snapshot-controller 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 175m
dns 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 174m
etcd 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 173m
image-registry 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 168m
ingress 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 167m
insights 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 169m
kube-apiserver 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 172m
kube-controller-manager 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 174m
kube-scheduler 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 173m
kube-storage-version-migrator 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 16m
machine-api 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 170m
machine-approver 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 173m
machine-config 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 172m
marketplace 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 174m
monitoring 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 167m
network 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 176m
node-tuning 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 9m23s
openshift-apiserver 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 170m
openshift-controller-manager 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 175m
openshift-samples 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 169m
operator-lifecycle-manager 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 175m
operator-lifecycle-manager-catalog 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 175m
operator-lifecycle-manager-packageserver 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 170m
service-ca 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 176m
storage 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest True False False 174m
melvinjoseph@mjoseph-mac tmp_dir % oc get co ingress -o json | jq .status.conditions
[
{
"lastTransitionTime": "2022-03-03T03:11:57Z",
"message": "The \"default\" ingress controller reports Available=True.",
"reason": "IngressAvailable",
"status": "True",
"type": "Available"
},
{
"lastTransitionTime": "2022-03-03T03:11:57Z",
"message": "desired and current number of IngressControllers are equal",
"reason": "AsExpected",
"status": "False",
"type": "Progressing"
},
{
"lastTransitionTime": "2022-03-03T03:11:57Z",
"message": "The \"default\" ingress controller reports Degraded=False.",
"reason": "IngressNotDegraded",
"status": "False",
"type": "Degraded"
},
{
"lastTransitionTime": "2022-03-03T05:36:08Z",
"reason": "IngressControllersUpgradeable",
"status": "True",
"type": "Upgradeable"
}
]
HENCE VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.9.26 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1022 |