Bug 2099668 - KubeControllerManager should degrade when GC stops working
Summary: KubeControllerManager should degrade when GC stops working
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 4.11
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.11.0
Assignee: Filip Krepinsky
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-21 13:09 UTC by Filip Krepinsky
Modified: 2022-08-10 11:18 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-10 11:18:49 UTC
Target Upstream Version:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-controller-manager-operator pull 623 0 None open Bug 2099668: introduce GC Watcher controller and add alerts for GarbageCollector 2022-06-21 13:12:45 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 11:18:58 UTC

Description Filip Krepinsky 2022-06-21 13:09:18 UTC 
Description of problem:

User should be notified when GC stops working because it cannot sync resources. 
Also observed in https://bugzilla.redhat.com/show_bug.cgi?id=2050912


How reproducible:


Steps to Reproduce:
- when resource discovery is not working properly / there are network disruptions
- or when some CRDs / APIServices are broken 

Actual results:
no reporting today unless we count KCM logs


Expected results:
KubeControllerManager should degrade and GarbageCollectorSyncFailed alert should be fired
Comment 1 Filip Krepinsky 2022-06-23 13:58:04 UTC 
I will add instruction on how to reproduce this next week
Comment 6 zhou ying 2022-06-28 09:53:52 UTC 
Checked with latest payload:

oc get clusterversion 
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-25-132614   True        False         115m    Error while reconciling 4.11.0-0.nightly-2022-06-25-132614: the cluster operator kube-controller-manager is degraded

Could see logs from KCM:
E0628 09:51:37.910863       1 shared_informer.go:258] unable to sync caches for garbage collector
E0628 09:51:37.910875       1 garbagecollector.go:245] timed out waiting for dependency graph builder sync during GC sync (attempt 176)

co degraded:
 oc get co  kube-controller-manager
NAME                      VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
kube-controller-manager   4.11.0-0.nightly-2022-06-25-132614   True        False         True       130m    GarbageCollectorDegraded: alerts firing: GarbageCollectorSyncFailed
Comment 8 errata-xmlrpc 2022-08-10 11:18:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069
Note You need to log in before you can comment on or make changes to this bug.