2060111 – ingress operator should report Upgradeable False to remind user before upgrade to 4.10 when Non-SAN certs are used

Bug 2060111 - ingress operator should report Upgradeable False to remind user before upgrade to 4.10 when Non-SAN certs are used

Summary: ingress operator should report Upgradeable False to remind user before upgrad...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.9
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	urgent
Target Milestone:	---
Target Release:	4.9.z
Assignee:	Miciah Dashiel Butler Masters
QA Contact:	Melvin Joseph
Docs Contact:
URL:
Whiteboard:
Depends On:	2059210
Blocks:
TreeView+	depends on / blocked

Reported:	2022-03-02 17:33 UTC by OpenShift BugZilla Robot
Modified:	2022-08-04 22:35 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: Users could configure an IngressController with a default certificate with a Common Name (CN) for the IngressController's domain but no Subject Alternative Name (SAN) for the same. Consequence: Clients built using Go 1.17 reject certificates without SANs. OpenShift 4.10 is built using Go 1.17, which means that various operators that connect to Routes that use the default certificate would reject the certificate and fail to complete the TLS handshake after upgrading to OpenShift 4.10. Fix: A check was added to the ingress operator, which will set Upgradeable=False on an IngressController if it has has a problematic default certificate. This status condition will propagate to the ingress ClusterOperator to block upgrading to 4.10. Result: OpenShift 4.9 blocks upgrades to 4.10 if an IngressController has a problematic default certificate.
Clone Of:
Environment:
Last Closed:	2022-03-29 07:16:22 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-ingress-operator pull 711	0	None	Merged	[release-4.9] Bug 2060111: Set Upgradeable=False if default cert has no SAN	2023-12-04 19:16:19 UTC
Red Hat Product Errata	RHBA-2022:1022	0	None	None	None	2022-03-29 07:16:38 UTC

Comment 1 Miciah Dashiel Butler Masters 2022-03-02 17:38:46 UTC

Setting blocker+ because we don't want to ship <https://github.com/openshift/cluster-ingress-operator/pull/709> without <https://github.com/openshift/cluster-ingress-operator/pull/711/commits/654c1a9f8a80cfcee59ecbe645764a292f0d0e5c> lest we break born-as-4.1 clusters.

Comment 2 Miciah Dashiel Butler Masters 2022-03-02 22:44:47 UTC

W. Trevor King pointed <https://github.com/openshift/cluster-ingress-operator/pull/711> out to me, which should prevent the issue with born-as-4.1 clusters by setting status.platformStatus in the cluster infrastructure config object on upgraded clusters.  I am therefore setting blocker- on this BZ.

Comment 3 Melvin Joseph 2022-03-03 06:05:26 UTC

melvinjoseph@mjoseph-mac Downloads % oc get clusterversion
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         106m    Cluster version is 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest
melvinjoseph@mjoseph-mac Downloads % mkdir test_customized_cert_no_san
melvinjoseph@mjoseph-mac Downloads % cd test_customized_cert_no_san 
melvinjoseph@mjoseph-mac test_customized_cert_no_san % ls
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl genrsa -out caKey.pem 2048
Generating RSA private key, 2048 bit long modulus
...........................+++
............................+++
e is 65537 (0x10001)
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl req -x509 -new -nodes -key caKey.pem -days 100000 -out caCert.pem -subj "/CN=network_edge_test_ca"
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl genrsa -out serverKey.pem 2048
Generating RSA private key, 2048 bit long modulus
....................+++
............................................................................................................................+++
e is 65537 (0x10001)
melvinjoseph@mjoseph-mac test_customized_cert_no_san % 
melvinjoseph@mjoseph-mac test_customized_cert_no_san % cat > server_no_san.conf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF
melvinjoseph@mjoseph-mac test_customized_cert_no_san % DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')
melvinjoseph@mjoseph-mac test_customized_cert_no_san % $DOMAIN
zsh: command not found: apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl req -new -key serverKey.pem -out serverNoSAN.csr -subj "/CN=*.$DOMAIN" -config server_no_san.conf
melvinjoseph@mjoseph-mac test_customized_cert_no_san % openssl x509 -req -in serverNoSAN.csr -CA caCert.pem -CAkey caKey.pem -CAcreateserial -out serverCertNoSAN.pem -days 100000 -extensions v3_req -extfile server_no_san.conf
Signature ok
subject=/CN=*.apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com
Getting CA Private Key
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc --namespace openshift-ingress create secret tls custom-certs-default --cert=serverCertNoSAN.pem --key=serverKey.pem
secret/custom-certs-default created
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
  --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'
ingresscontroller.operator.openshift.io/default patched
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc create configmap user-ca-bundle --from-file=ca-bundle.crt=caCert.pem -n openshift-config
configmap/user-ca-bundle created
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc patch proxy/cluster --patch '{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}' --type=merge
proxy.config.openshift.io/cluster patched
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc get co                                        
NAME                                       VERSION                                                  AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      18m     
baremetal                                  4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      145m    
cloud-controller-manager                   4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      148m    
cloud-credential                           4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      151m    
cluster-autoscaler                         4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      145m    
config-operator                            4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      146m    
console                                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      19m     
csi-snapshot-controller                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      146m    
dns                                        4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      144m    
etcd                                       4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      144m    
image-registry                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      139m    
ingress                                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      138m    
insights                                   4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      140m    
kube-apiserver                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      143m    
kube-controller-manager                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      144m    
kube-scheduler                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      144m    
kube-storage-version-migrator              4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      146m    
machine-api                                4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      141m    
machine-approver                           4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      144m    
machine-config                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      143m    
marketplace                                4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      145m    
monitoring                                 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      137m    
network                                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      147m    
node-tuning                                4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      118s    
openshift-apiserver                        4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      140m    
openshift-controller-manager               4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      145m    
openshift-samples                          4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      140m    
operator-lifecycle-manager                 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      146m    
operator-lifecycle-manager-catalog         4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      146m    
operator-lifecycle-manager-packageserver   4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      141m    
service-ca                                 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      146m    
storage                                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      145m    
melvinjoseph@mjoseph-mac test_customized_cert_no_san % oc get co ingress -o json | jq .status.conditions
[
  {
    "lastTransitionTime": "2022-03-03T03:11:57Z",
    "message": "The \"default\" ingress controller reports Available=True.",
    "reason": "IngressAvailable",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2022-03-03T03:11:57Z",
    "message": "desired and current number of IngressControllers are equal",
    "reason": "AsExpected",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2022-03-03T03:11:57Z",
    "message": "The \"default\" ingress controller reports Degraded=False.",
    "reason": "IngressNotDegraded",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2022-03-03T05:10:55Z",
    "message": "Some ingresscontrollers are not upgradeable: ingresscontroller \"default\" is not upgradeable: OperandsNotUpgradeable: One or more managed resources are not upgradeable: certificate in secret openshift-ingress/custom-certs-default has legacy Common Name (CN) but has no Subject Alternative Name (SAN) for domain: *.apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com",
    "reason": "IngressControllersNotUpgradeable",
    "status": "False",
    "type": "Upgradeable"
  }
]


FIRST PART IS VERIFIED.


melvinjoseph@mjoseph-mac Downloads % mkdir tmp_dir
melvinjoseph@mjoseph-mac Downloads % cd tmp_dir 
melvinjoseph@mjoseph-mac tmp_dir % curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/ca.key
melvinjoseph@mjoseph-mac tmp_dir % curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/ca.pem
melvinjoseph@mjoseph-mac tmp_dir %  curl -O -sS https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/openssl.conf
melvinjoseph@mjoseph-mac tmp_dir % $DOMAIN
zsh: command not found: apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com
melvinjoseph@mjoseph-mac tmp_dir % sed -i.bak "s/example.com/${DOMAIN}/g" openssl.conf
melvinjoseph@mjoseph-mac tmp_dir % LS
ca.key			ca.pem			openssl.conf		openssl.conf.bak
melvinjoseph@mjoseph-mac tmp_dir % openssl genrsa -out apps.key 2048
Generating RSA private key, 2048 bit long modulus
....................................+++
.............................+++
e is 65537 (0x10001)
melvinjoseph@mjoseph-mac tmp_dir % openssl req -new -config openssl.conf -key apps.key -out apps.csr
melvinjoseph@mjoseph-mac tmp_dir % openssl x509 -req -CA ca.pem -CAkey ca.key -CAcreateserial -extfile openssl.conf -extensions v3_req -in apps.csr -out apps.crt -days 3650
Signature ok
subject=/C=US/ST=VA/L=Somewhere/O=RedHat/OU=OpenShift QE/CN=apps
Getting CA Private Key
melvinjoseph@mjoseph-mac tmp_dir % openssl x509 -text -noout -in apps.crt | grep "Alternative Name" -A 1
            X509v3 Subject Alternative Name: 
                DNS:*.apps.ci-ln-pz9dkbk-72292.origin-ci-int-gce.dev.rhcloud.com
melvinjoseph@mjoseph-mac tmp_dir % oc --namespace openshift-ingress create secret tls custom-certs-default-new --cert=apps.crt --key=apps.key        
secret/custom-certs-default-new created
melvinjoseph@mjoseph-mac tmp_dir % oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
  --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default-new"}}}'
ingresscontroller.operator.openshift.io/default patched
melvinjoseph@mjoseph-mac tmp_dir % oc create configmap user-ca-bundle2 --from-file=ca-bundle.crt=ca.pem -n openshift-config 
configmap/user-ca-bundle2 created
melvinjoseph@mjoseph-mac tmp_dir % oc patch proxy/cluster --patch '{"spec":{"trustedCA":{"name":"user-ca-bundle2"}}}' --type=merge
proxy.config.openshift.io/cluster patched
melvinjoseph@mjoseph-mac tmp_dir % oc -n openshift-ingress get secret
NAME                           TYPE                                  DATA   AGE
builder-dockercfg-szphp        kubernetes.io/dockercfg               1      148m
builder-token-dpp78            kubernetes.io/service-account-token   4      148m
builder-token-hnk59            kubernetes.io/service-account-token   4      148m
custom-certs-default           kubernetes.io/tls                     2      26m
custom-certs-default-new       kubernetes.io/tls                     2      71s
default-dockercfg-4p2wd        kubernetes.io/dockercfg               1      148m
default-token-96crm            kubernetes.io/service-account-token   4      148m
default-token-s6s2x            kubernetes.io/service-account-token   4      148m
deployer-dockercfg-g2p5b       kubernetes.io/dockercfg               1      148m
deployer-token-9np2n           kubernetes.io/service-account-token   4      148m
deployer-token-x44wx           kubernetes.io/service-account-token   4      148m
router-dockercfg-qdb9f         kubernetes.io/dockercfg               1      148m
router-metrics-certs-default   kubernetes.io/tls                     2      148m
router-stats-default           Opaque                                2      148m
router-token-9nsp5             kubernetes.io/service-account-token   4      148m
router-token-pps4x             kubernetes.io/service-account-token   4      148m

melvinjoseph@mjoseph-mac tmp_dir % oc get co
NAME                                       VERSION                                                  AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      47m     
baremetal                                  4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      174m    
cloud-controller-manager                   4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      177m    
cloud-credential                           4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      3h      
cluster-autoscaler                         4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      174m    
config-operator                            4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      176m    
console                                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      22m     
csi-snapshot-controller                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      175m    
dns                                        4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      174m    
etcd                                       4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      173m    
image-registry                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      168m    
ingress                                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      167m    
insights                                   4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      169m    
kube-apiserver                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      172m    
kube-controller-manager                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      174m    
kube-scheduler                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      173m    
kube-storage-version-migrator              4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      16m     
machine-api                                4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      170m    
machine-approver                           4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      173m    
machine-config                             4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      172m    
marketplace                                4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      174m    
monitoring                                 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      167m    
network                                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      176m    
node-tuning                                4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      9m23s   
openshift-apiserver                        4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      170m    
openshift-controller-manager               4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      175m    
openshift-samples                          4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      169m    
operator-lifecycle-manager                 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      175m    
operator-lifecycle-manager-catalog         4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      175m    
operator-lifecycle-manager-packageserver   4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      170m    
service-ca                                 4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      176m    
storage                                    4.9.0-0.ci.test-2022-03-03-025003-ci-ln-pz9dkbk-latest   True        False         False      174m    
melvinjoseph@mjoseph-mac tmp_dir % oc get co ingress -o json | jq .status.conditions
[
  {
    "lastTransitionTime": "2022-03-03T03:11:57Z",
    "message": "The \"default\" ingress controller reports Available=True.",
    "reason": "IngressAvailable",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2022-03-03T03:11:57Z",
    "message": "desired and current number of IngressControllers are equal",
    "reason": "AsExpected",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2022-03-03T03:11:57Z",
    "message": "The \"default\" ingress controller reports Degraded=False.",
    "reason": "IngressNotDegraded",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2022-03-03T05:36:08Z",
    "reason": "IngressControllersUpgradeable",
    "status": "True",
    "type": "Upgradeable"
  }
]

HENCE VERIFIED

Comment 8 errata-xmlrpc 2022-03-29 07:16:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.26 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1022

Note You need to log in before you can comment on or make changes to this bug.