Bug 2099668

Summary:	KubeControllerManager should degrade when GC stops working
Product:	OpenShift Container Platform	Reporter:	Filip Krepinsky <fkrepins>
Component:	kube-controller-manager	Assignee:	Filip Krepinsky <fkrepins>
Status:	CLOSED ERRATA	QA Contact:	zhou ying <yinzhou>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.11	CC:	mfojtik
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 11:18:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Filip Krepinsky 2022-06-21 13:09:18 UTC

Description of problem:

User should be notified when GC stops working because it cannot sync resources. 
Also observed in https://bugzilla.redhat.com/show_bug.cgi?id=2050912


How reproducible:


Steps to Reproduce:
- when resource discovery is not working properly / there are network disruptions
- or when some CRDs / APIServices are broken 

Actual results:
no reporting today unless we count KCM logs


Expected results:
KubeControllerManager should degrade and GarbageCollectorSyncFailed alert should be fired

Comment 1 Filip Krepinsky 2022-06-23 13:58:04 UTC

I will add instruction on how to reproduce this next week

Comment 6 zhou ying 2022-06-28 09:53:52 UTC

Checked with latest payload:

oc get clusterversion 
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-25-132614   True        False         115m    Error while reconciling 4.11.0-0.nightly-2022-06-25-132614: the cluster operator kube-controller-manager is degraded

Could see logs from KCM:
E0628 09:51:37.910863       1 shared_informer.go:258] unable to sync caches for garbage collector
E0628 09:51:37.910875       1 garbagecollector.go:245] timed out waiting for dependency graph builder sync during GC sync (attempt 176)

co degraded:
 oc get co  kube-controller-manager
NAME                      VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
kube-controller-manager   4.11.0-0.nightly-2022-06-25-132614   True        False         True       130m    GarbageCollectorDegraded: alerts firing: GarbageCollectorSyncFailed

Comment 8 errata-xmlrpc 2022-08-10 11:18:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069