Bug 1775253 - Metrics endpoints of catalog-operator and olm-operator are potentially broken by service CA rotation
Summary: Metrics endpoints of catalog-operator and olm-operator are potentially broken...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.1.z
Assignee: Jeff Peeler
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1775250
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-21 15:39 UTC by Nick Hale
Modified: 2020-01-09 09:16 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1771811
Environment:
Last Closed: 2020-01-09 09:16:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1172 0 None closed bug 1775253: make certificate updates live upon update 2020-04-08 03:37:47 UTC
Red Hat Product Errata RHBA-2020:0010 0 None None None 2020-01-09 09:16:28 UTC

Comment 2 Jian Zhang 2019-12-24 09:27:35 UTC
Cluster version is 4.1.0-0.nightly-2019-12-23-102617
mac:~ jianzhang$ oc exec catalog-operator-785c88dd64-64zvw -- olm --version
OLM version: 0.9.0
git commit: 23f5c0292434c41743676abdd2da88f6a26990f5

1, Save the original cert info.
mac:~ jianzhang$ oc port-forward catalog-operator-785c88dd64-64zvw 8081:8081
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Handling connection for 8081

mac:~ jianzhang$ echo | openssl s_client -connect localhost:8081 2>&1 | gsed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > olm1224.crt

2, Delete secret catalog-operator-serving-cert   olm-operator-serving-cert 
mac:~ jianzhang$ oc get secret
NAME                                             TYPE                                  DATA   AGE
builder-dockercfg-gwk4v                          kubernetes.io/dockercfg               1      26m
builder-dockercfg-k7djp                          kubernetes.io/dockercfg               1      26m
builder-token-f5zhq                              kubernetes.io/service-account-token   4      27m
builder-token-fdwpq                              kubernetes.io/service-account-token   4      28m
catalog-operator-serving-cert                    kubernetes.io/tls                     2      29s
default-dockercfg-k8wd7                          kubernetes.io/dockercfg               1      26m
default-token-2kvwm                              kubernetes.io/service-account-token   4      32m
default-token-mqjgk                              kubernetes.io/service-account-token   4      26m
deployer-dockercfg-k89hg                         kubernetes.io/dockercfg               1      26m
deployer-token-4nhqb                             kubernetes.io/service-account-token   4      28m
deployer-token-rmwl8                             kubernetes.io/service-account-token   4      27m
olm-operator-serviceaccount-dockercfg-t8546      kubernetes.io/dockercfg               1      26m
olm-operator-serviceaccount-token-tkmqk          kubernetes.io/service-account-token   4      32m
olm-operator-serviceaccount-token-zr68j          kubernetes.io/service-account-token   4      26m
olm-operator-serving-cert                        kubernetes.io/tls                     2      29s
olm-operators-configmap-server-dockercfg-jqk5s   kubernetes.io/dockercfg               1      26m
olm-operators-configmap-server-token-hrtcc       kubernetes.io/service-account-token   4      26m
olm-operators-configmap-server-token-zvsts       kubernetes.io/service-account-token   4      30m
packageserver-dockercfg-5dcg5                    kubernetes.io/dockercfg               1      28m
packageserver-token-hdmvk                        kubernetes.io/service-account-token   4      28m
packageserver-token-smbf4                        kubernetes.io/service-account-token   4      28m
v1.packages.operators.coreos.com-cert            kubernetes.io/tls                     2      28m

3, After they are recreated, save the cert info again.
mac:~ jianzhang$ oc port-forward catalog-operator-785c88dd64-64zvw 8081:8081
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Handling connection for 8081
...
mac:~ jianzhang$ echo | openssl s_client -connect localhost:8081 2>&1 | gsed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > olm1224-3.crt

4, Check if the cert info are different. See below, they are different. LGTM, verify it.
mac:~ jianzhang$ diff olm1224.crt olm1224-3.crt 
2c2
< MIIEFjCCAv6gAwIBAgIIXjybi4qtiOswDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
---
> MIIEFjCCAv6gAwIBAgIIfZe2NZTj1wQwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
4c4
< Fw0xOTEyMjQwODQ5NDFaFw0yMTEyMjMwODQ5NDJaMEwxSjBIBgNVBAMTQWNhdGFs
---
> Fw0xOTEyMjQwOTIwMjlaFw0yMTEyMjMwOTIwMzBaMEwxSjBIBgNVBAMTQWNhdGFs
6,12c6,12
< LW1hbmFnZXIuc3ZjMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAt53a
< mVSbExFzcdyJpD3Eipgh/ghjk1MNNQSmVCHE4PAVq61RAJ4yiOySXnl0anmq/S8u
< AT8Gmae99PCsxeBPCsEUgomUB02ZdLOa+h4cV/Y8JsLhFlXXRLfkuw2H2p3mNISW
< v9IBGDg8UL4tFbiOb4msAyJZutt/XNjJg8uxq9D5IZG6S1FoQWeUB6W/VEtVR8dB
< 6jCMUHjDfPNATtpa7ydcDj0ptZFLZbVMEJjwdShbEtlQKT4Jj1j+QpeCoMrtMxiu
< dRnSc+GmnkV6p+mIjZ4CtvCaVRj5T2vpbJM9aseQa6jK/3r7Q2pdaeVhfVCelLey
< p1kfp8E7Rp+b2dFwJwIDAQABo4IBEDCCAQwwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud
---
> LW1hbmFnZXIuc3ZjMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA19tz
> EtYWneNcPAiB8gKUCRBoUOADLBv7W1ltLFO/ic0JQ/CfkNzn9QWmBFF4On6aUlXv
> pDF2Zl3R78tOV6UinpR5JUNPBjeliCA7cT3uiz5kxpLMWmn1A4s0RM5uRLA0ZrVw
> hUrtndrieFvHUr02bHeBlkfr4pJQ3XP0stgE85MDlvUSzpoYZ5pY6VQaz6EYrqHC
> FMq/op68Xq6uxYQUdeAtJvJLSEDN6+VbiqzqD26UYX2bKy/yHfQxPgVEV00soGNr
> CBGx8melnZdolLQWenbKRkN1ycuGaGXZeBnM3PojThxGfoNmX3IoyS6vrdytNlZ4
> fx1IQvlnUdOmRgue6QIDAQABo4IBEDCCAQwwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud
18,23c18,23
< NWQ0ZDY3MA0GCSqGSIb3DQEBCwUAA4IBAQB8ahrL+cqVY/OQEjIt9DBsRAZdBWtO
< 0zwvviDZUpjfE8a1e5ms4H5qHgiP6w2SXznnMFlOAQVquPkTzW7Dbgd8mL/fhMzk
< inbAQEoJRDUDiW+jAo8pwIfls6QdHF9oYTSfulNmtQhyh4Z+HFGX1xMKNC0YVU8b
< zPJlZvrcrEJ1iFzOk6F+sNI1aeNJr1a47CxMcci7BPeU+W8PLwtSg7Z1/OBn1Loe
< vi0K5Nmix/oD8EGYFA8SAaKeRF4iBYf0twNdrQUmC/4s8+ubN6XHGVfi6tpEfEQn
< zHSth6dlmZPi1XT+lzVoM1OlmqDSyE88eYogzRcyt17D8Q+fzG+tXJN7
---
> NWQ0ZDY3MA0GCSqGSIb3DQEBCwUAA4IBAQB5AdkjcAui32CQ6W+GUqcRMomzv3Ie
> A3vYT6BqGafOO65/UUdCh0KDb0RDIFRawpBu/whd5OJRzyVRK+f5SG6dUMRsPQaO
> qQHyhLH3v5pdK95lqotjfi3HIrS3ShZ1IkX3x+Dt8F+OPrcY6BTemzJ0bPmGbu31
> 6ZImHIiE5tNZTqKpKOInkwV/9woqio1mJPLGRLoXGfpHg7FrqHI69XT+l3Ljt5qd
> SF06Bywrfk1yytaqW9+jc72XAJNBTedLVzjEw7y7vpQy185E4ewpsU8Ee6a108QY
> ZqBg7MUBxEHtsBE61KX4OV9phQMFKNu7zzruKN7nnxyRMNHk4TZrYmSh

Comment 4 errata-xmlrpc 2020-01-09 09:16:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0010


Note You need to log in before you can comment on or make changes to this bug.