Bug 1821690

Summary: When changed CSR signer hot loops on update
Product: OpenShift Container Platform Reporter: Tomáš Nožička <tnozicka>
Component: kube-controller-managerAssignee: Tomáš Nožička <tnozicka>
Status: CLOSED ERRATA QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: high    
Version: 4.4CC: aos-bugs, maszulik, mfojtik, yinzhou
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1821689 Environment:
Last Closed: 2020-05-04 11:48:34 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1821689    
Bug Blocks:    

Description Tomáš Nožička 2020-04-07 12:24:59 UTC
+++ This bug was initially created as a clone of Bug #1821689 +++

CKCMO keeps updating the csr-signer although no change is needed.

I0407 10:54:35.081638  298746 csrcontroller.go:167] CSRController sync done
I0407 10:54:35.081699  298746 csrcontroller.go:128] Starting CSRController sync
I0407 10:54:35.081791  298746 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-controller-manager-operator", Name:"kube-controller-manager-operator", UID:"840796e5-22d8-479c-b539-05c11be3f958", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/csr-signer -n openshift-kube-controller-manager because it changed
I0407 10:54:46.046043  298746 core.go:281] Secret openshift-kube-controller-manager/csr-signer changes: {"type":null}
I0407 10:54:46.230221  298746 csrcontroller.go:164] Refreshed CSRSigner.
I0407 10:54:46.230237  298746 csrcontroller.go:167] CSRController sync done
I0407 10:54:46.230250  298746 csrcontroller.go:128] Starting CSRController sync
I0407 10:54:46.230249  298746 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-controller-manager-operator", Name:"kube-controller-manager-operator", UID:"840796e5-22d8-479c-b539-05c11be3f958", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/csr-signer -n openshift-kube-controller-manager because it changed
I0407 10:54:57.144311  298746 core.go:281] Secret openshift-kube-controller-manager/csr-signer changes: {"type":null}
I0407 10:54:57.323193  298746 csrcontroller.go:164] Refreshed CSRSigner.
I0407 10:54:57.323220  298746 csrcontroller.go:167] CSRController sync done

Comment 3 zhou ying 2020-04-13 06:29:19 UTC
Confirmed with payload :4.4.0-0.nightly-2020-04-09-220855, the issue has fixed:
1) one terminal delete secrets :

`oc delete secrets csr-signer -n  openshift-kube-controller-manager`


2) on second terminal check logs from CKCMO:
oc logs -f po/kube-controller-manager-operator-5f47c4d756-l8p8p  -n openshift-kube-controller-manager-operator

I0413 01:57:25.894786       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-controller-manager-operator", Name:"kube-controller-manager-operator", UID:"22273bd2-2b95-426d-93bc-06f3191ab756", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nStaticPodsDegraded: nodes/ip-10-0-163-185.us-east-2.compute.internal pods/kube-controller-manager-ip-10-0-163-185.us-east-2.compute.internal container=\"cluster-policy-controller\" is not ready" to "NodeControllerDegraded: All master nodes are ready"





I0413 06:26:54.336426       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-controller-manager-operator", Name:"kube-controller-manager-operator", UID:"22273bd2-2b95-426d-93bc-06f3191ab756", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/csr-signer -n openshift-kube-controller-manager because it was missing

Comment 5 errata-xmlrpc 2020-05-04 11:48:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581