Bug 1711185

Summary: [DR][MSTR-363] Run regenerate-certificates command failed when do the certificate recovery
Product: OpenShift Container Platform Reporter: zhou ying <yinzhou>
Component: MasterAssignee: Tomáš Nožička <tnozicka>
Status: CLOSED ERRATA QA Contact: zhou ying <yinzhou>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.1.0CC: aos-bugs, bleanhar, jokerman, mmccomas, tnozicka, wsun
Target Milestone: ---Keywords: Regression
Target Release: 4.1.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:48:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description zhou ying 2019-05-17 07:41:28 UTC
Description of problem:
Run the regenerate-certificates command on master failed with error:
E0517 02:57:08.238044       1 reflector.go:134] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: Failed to list *v1.Infrastructure: the server could not find the requested resource (get infrastructures.config.openshift.io)
E0517 02:57:08.240989       1 reflector.go:134] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: Failed to list *v1.Network: the server could not find the requested resource (get networks.config.openshift.io)

Version-Release number of selected component (if applicable):
Payload: 4.1.0-0.nightly-2019-05-16-075717 or later

How reproducible:
Always

Steps to Reproduce:
1. Follow the doc: https://docs.google.com/document/d/1ONkxdDmQVLBNJrSJymfKPrndo7b4vgCA2zwL9xHYx6A/edit to do certificate recovery;


Actual results:
1.  Failed when run regenerate-certificates command on master:
[root@ip-10-0-131-147 ~]# podman run -it --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator "${KAO_IMAGE}" regenerate-certificates
I0517 02:57:08.210033       1 certrotationcontroller.go:452] Waiting for CertRotation
E0517 02:57:08.238044       1 reflector.go:134] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: Failed to list *v1.Infrastructure: the server could not find the requested resource (get infrastructures.config.openshift.io)
E0517 02:57:08.240989       1 reflector.go:134] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: Failed to list *v1.Network: the server could not find the requested resource (get networks.config.openshift.io)
......

Expected results:
1. Should succeed.

Additional info:
Older payload: 4.1.0-0.nightly-2019-05-15-151517 does not have this issue.

Comment 4 zhou ying 2019-05-20 09:59:59 UTC
Confirmed with the payload: 4.1.0-0.nightly-2019-05-18-050636, the issue has fixed. 

[root@ip-10-0-128-98 ~]# podman run -it --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator "${KAO_IMAGE}" regenerate-certificates
I0520 01:46:38.431255       1 certrotationcontroller.go:452] Waiting for CertRotation
I0520 01:46:38.531536       1 client_cert_rotation_controller.go:117] Waiting for CertRotationController - "AggregatorProxyClientCert"
I0520 01:46:38.631711       1 client_cert_rotation_controller.go:124] Finished waiting for CertRotationController - "AggregatorProxyClientCert"
......
I0520 01:46:58.314770       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/service-network-serving-certkey/tls.crt"
I0520 01:46:58.314938       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/service-network-serving-certkey/tls.key"
I0520 01:46:58.319064       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/client-ca/ca-bundle.crt"
I0520 01:46:58.322952       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/localhost-serving-cert-certkey/tls.crt"
I0520 01:46:58.323125       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/localhost-serving-cert-certkey/tls.key"
I0520 01:46:58.328699       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key"
I0520 01:46:58.328909       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt"
I0520 01:46:58.332729       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8/secrets/kube-controller-manager-client-cert-key/tls.crt"
I0520 01:46:58.332887       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8/secrets/kube-controller-manager-client-cert-key/tls.key"
I0520 01:46:58.336739       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8/secrets/csr-signer/tls.crt"
I0520 01:46:58.337222       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8/secrets/csr-signer/tls.key"
I0520 01:46:58.340512       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-scheduler-pod-7/secrets/kube-scheduler-client-cert-key/tls.crt"
I0520 01:46:58.340681       1 helpers.go:121] Wrote new content to file "/etc/kubernetes/static-pod-resources/kube-scheduler-pod-7/secrets/kube-scheduler-client-cert-key/tls.key"

Comment 6 errata-xmlrpc 2019-06-04 10:48:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758