Bug 1775250

Summary:	Metrics endpoints of catalog-operator and olm-operator are potentially broken by service CA rotation
Product:	OpenShift Container Platform	Reporter:	Nick Hale <nhale>
Component:	OLM	Assignee:	Jeff Peeler <jpeeler>
OLM sub component:	OLM	QA Contact:	Jian Zhang <jiazha>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	unspecified	CC:	jiazha, jpeeler, mnewby, nhale
Version:	4.2.z
Target Milestone:	---
Target Release:	4.2.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1771811	Environment:
Last Closed:	2020-01-07 17:55:10 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1777593
Bug Blocks:	1775253

Comment 2 Jian Zhang 2019-12-17 06:36:34 UTC

Cluster version is 4.2.0-0.nightly-2019-12-15-230238
mac:~ jianzhang$ oc exec catalog-operator-67687b4458-mw8gs -- olm --version
OLM version: 0.11.0
git commit: 68a313001672be96bd094840839f9e755e9b9606

mac:~ jianzhang$ oc get pods
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-67687b4458-mw8gs   1/1     Running   0          31m
olm-operator-67d4486cd4-gjsx5       1/1     Running   0          31m
packageserver-5cf7589445-5wk2p      1/1     Running   0          29m
packageserver-5cf7589445-xs6vb      1/1     Running   0          29m

mac:~ jianzhang$ oc port-forward catalog-operator-67687b4458-mw8gs  8081:8081
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Handling connection for 8081

1, Open another terminal to save the cert info.
mac:~ jianzhang$ echo | openssl s_client -connect localhost:8081 2>&1 | gsed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > bug.crt
mac:~ jianzhang$ 

2, Delete the secret catalog-operator-serving-cert and olm-operator-serving-cert. And, wait for they are rebuilt.
mac:~ jianzhang$ oc get secret
NAME                                          TYPE                                  DATA   AGE
builder-dockercfg-w9w6v                       kubernetes.io/dockercfg               1      27m
builder-token-llmqm                           kubernetes.io/service-account-token   4      27m
builder-token-sv9q4                           kubernetes.io/service-account-token   4      28m
catalog-operator-serving-cert                 kubernetes.io/tls                     2      8s
default-dockercfg-qrmsg                       kubernetes.io/dockercfg               1      27m
default-token-n24qm                           kubernetes.io/service-account-token   4      28m
default-token-ttc7f                           kubernetes.io/service-account-token   4      33m
deployer-dockercfg-q75hp                      kubernetes.io/dockercfg               1      27m
deployer-token-5q7s9                          kubernetes.io/service-account-token   4      27m
deployer-token-v7rmt                          kubernetes.io/service-account-token   4      28m
olm-operator-serviceaccount-dockercfg-n8p7l   kubernetes.io/dockercfg               1      27m
olm-operator-serviceaccount-token-tmlcz       kubernetes.io/service-account-token   4      33m
olm-operator-serviceaccount-token-zc4jf       kubernetes.io/service-account-token   4      27m
olm-operator-serving-cert                     kubernetes.io/tls                     2      7s
v1.packages.operators.coreos.com-cert         kubernetes.io/tls                     2      31m

3, Save the new cert. 
mac:~ jianzhang$ oc port-forward catalog-operator-67687b4458-mw8gs  8081:8081
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Handling connection for 8081

mac:~ jianzhang$ echo | openssl s_client -connect localhost:8081 2>&1 | gsed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > bug2.crt
mac:~ jianzhang$ 

4, Compare the two certs.
mac:~ jianzhang$ diff bug.crt bug2.crt 
2c2
< MIIEFjCCAv6gAwIBAgIIPsKClSdnHFEwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
---
> MIIEFjCCAv6gAwIBAgIILsIgsgiAR+MwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
4c4
< Fw0xOTEyMTcwNTU3NDVaFw0yMTEyMTYwNTU3NDZaMEwxSjBIBgNVBAMTQWNhdGFs
---
> Fw0xOTEyMTcwNjI5NDdaFw0yMTEyMTYwNjI5NDhaMEwxSjBIBgNVBAMTQWNhdGFs
6,12c6,12
< LW1hbmFnZXIuc3ZjMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApUwU
< X5zpKRIYEylKerL+MXgImcrswemKIVU9SBtPqsmkuz1Q6g1lXrrGEe2iS+f07P3E
< YeCOB5Y52sJks004hLtQ0RMOL6mg7QnxvNvF2Dm+RU4d+1YnIZF0S1Va/0/Lsj06
< ndmtz2ldSFaECFoLO3FCQq5C0U0ZKDx7SJsDlhZWBsbCmwkPOHx48g3QhlIqjP+2
< FfPieIE0uY4pbYxmqN7x2lYKXfNixhL6DmDmmqzBxalynsekam0EyLJtYCqtYLi+
< Qbj4dC2W/uQNFQbiKEwMAKjHs2kPNQH27FkedDO/5QKcgR8VvRuNfd/C3ztYZ25g
< k81TWbUphDBEfYBd5wIDAQABo4IBEDCCAQwwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud
---
> LW1hbmFnZXIuc3ZjMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4krp
> hmDIy61FiweWXcHUNrMLYFhcB6fHBP5WEiiwKft85Vz7PVY5aT6xOa2nEjB1cFXs
> KMG4UwQOPlwkPAY81dESw0Rk4Oy+/eYbNy81xuxSTfTT0Rn3Q5koJ24r+aPApKPc
> B2cRn4s50QyWV1K3+0pGmQxWYZtUsh1lFFeROFtvI3igH3h9m8PqZ5wtnwRTB8++
> QGB2d19NsFGcep+QPFblcA6qd372oWE8G39mCCKmUWAfsHepQH/XKXpY4viHeAyP
> gCpLuWkUV39nU0bjAIF5OD4fSc0KjPF1HXj9bEzJDSFm0o1KqntMvNpIKSxbnabN
> rCBMOMagFS3I3y6DXwIDAQABo4IBEDCCAQwwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud
18,23c18,23
< YWQ1YTkzMA0GCSqGSIb3DQEBCwUAA4IBAQBLCi9+tsb3UERjejn/jXoOV/urML74
< GzHM2ngyHNufv5Z47vQbJ6iWF7EU/3plTr+mkLbu7FQxKAYITj+3YNUdzRg6G+z6
< JRUVeilx0eFBgbq7nXyQJ5kZHn1/e9YtiV4wRftJaByvX60JYMuB7SneQCq922gH
< wJxvohTuoTSvtnyk+W6QKyT+M1wCafTEzjh8zgWOFIGPJAziQTZcem21EpXg5EOZ
< IYGUynqZRuuFn3Tg/bisQolSbhBcvRG7C8zrDD3GlSRbiESfE03MciPHxtT7FQMr
< ec1ec9wAAEbgyLDkyvuzbFsdCVj/KvsNJ6YlB026OQ3bYQZ9Y1/1U4LI
---
> YWQ1YTkzMA0GCSqGSIb3DQEBCwUAA4IBAQCvWl0prgSsCGUWpKcRf5+NVx8VTUn6
> 4FYotLwkVMDsTE6M8X9iKaUpcdoTAXXhrlTsDKL2LUA40UQ4av4pWA3/UNLdpXSD
> 2apBMC1NMG7boA3IiShxNuxwm+FalSaRycwwiTF7hi6+SypebpFBCIISDBRrVd2g
> paTB7qb0AW0Im0nTJJlnfQCE7nti3QYDYwjQhyNQEwl1V70hHFV1QdXRthN1ICxo
> DeUa9lnzZPcaWt/K21nU2bQG1QAFsA9ixbBz/NQaa/v8UcfRfyZ9AU7/PwUQJxpY
> VcptIC9AalkO0qwf2AQmEaamC5mX/ASLy6kLSTAfdKHjCP0EEaa3ON98

They are different, LGTM, verify it.

Comment 4 errata-xmlrpc 2020-01-07 17:55:10 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0014