Bug 1820508

Summary:	MCO creates thousands of CSRs overnight
Product:	OpenShift Container Platform	Reporter:	Tomáš Nožička <tnozicka>
Component:	Node	Assignee:	Ryan Phillips <rphillips>
Status:	CLOSED DUPLICATE	QA Contact:	Sunil Choudhary <schoudha>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.4	CC:	aos-bugs, jokerman, mpatel, rphillips
Target Milestone:	---
Target Release:	4.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-04-07 18:48:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Tomáš Nožička 2020-04-03 08:57:01 UTC

Description of problem:
MCO hotloops on creating CSRs after the cluster has been shutdown for 25 h and in i process of recovery.

$ LANG=en date && oc get csr | grep system:serviceaccount:openshift-machine-config-operator:node-bootstrapper | wc -l
Fri Apr  3 10:51:01 CEST 2020
3404

$ LANG=en date && oc get csr | grep system:serviceaccount:openshift-machine-config-operator:node-bootstrapper | wc -l
Fri Apr  3 10:52:17 CEST 2020
3414



Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-04-01-141451


How reproducible:

Steps to Reproduce:
1. shutdown the cluster for 25 h, or ping tnozicka (I may still have the one that's broken)


Actual results:
thousands of CSRs, new ones at rate about 10 per minute


Expected results:
Only 1 CSR is created and it stays Pending until the admin approves it.


Additional info:

Comment 1 Antonio Murdaca 2020-04-03 09:00:14 UTC

Ryan, has something changed here?

Comment 2 Tomáš Nožička 2020-04-03 09:03:14 UTC

is kubelet or something else using the same SA? the machine-config-operator pod is dead when I looked on the node with crictl

Comment 3 Antonio Murdaca 2020-04-03 09:11:03 UTC

(In reply to Tomáš Nožička from comment #2)
> is kubelet or something else using the same SA? the machine-config-operator
> pod is dead when I looked on the node with crictl

can you grab must-gather meanwhile, it'll help whoever will debug this.

Comment 4 Tomáš Nožička 2020-04-03 09:44:33 UTC

I can't, must-gather requires running pods. Also pod logs are not working without valid certs.

Comment 5 Tomáš Nožička 2020-04-06 15:34:21 UTC

kubelet was restarting because of another bug (being fixed now) and creating new CSR every time, although it had one already pending. Given this comes from upstream and with the fatal bug now being fixed I am lowering the severity and sending it to Node team to decide if they want to pursue, close or convert to Jira card.

Comment 6 Ryan Phillips 2020-04-07 18:48:32 UTC

Fixed via https://github.com/openshift/origin/pull/24801 and BZ 1818961

*** This bug has been marked as a duplicate of bug 1818961 ***