Bug 2103940 - kube-controller-manager operator 4.11.0-rc.0 degraded on disabled monitoring stack
Summary: kube-controller-manager operator 4.11.0-rc.0 degraded on disabled monitoring ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 4.11
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.12.0
Assignee: Filip Krepinsky
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks: 2104552
TreeView+ depends on / blocked
 
Reported: 2022-07-05 11:06 UTC by Adrian
Modified: 2023-01-17 19:51 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Duplication of https://bugzilla.redhat.com/show_bug.cgi?id=2118286
Clone Of:
Environment:
Last Closed: 2023-01-17 19:51:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
must-gather (6.71 MB, application/gzip)
2022-07-05 11:06 UTC, Adrian 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-controller-manager-operator pull 636 0 None open Bug 2103940: Decouple KCM-O's status from monitoring stack 2022-07-06 13:42:54 UTC
Red Hat Product Errata RHSA-2022:7399 0 None None None 2023-01-17 19:51:44 UTC

Description Adrian 2022-07-05 11:06:52 UTC 
Created attachment 1894658 [details]
must-gather

Description of problem:

Created a single node cluster with monitoring stack disabled, now the kube-controller-manager operator shows degraded state due to dependency with thanos-querier 

Version-Release number of selected component (if applicable):

4.11.0-rc.0


Actual results:

```bash
[cloud-user@rhel-crcqe /]$ oc get co
NAME                                       VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.11.0-rc.0   True        False         False      65m     
config-operator                            4.11.0-rc.0   True        False         False      46h     
console                                    4.11.0-rc.0   True        False         False      66m     
dns                                        4.11.0-rc.0   True        False         False      71m     
etcd                                       4.11.0-rc.0   True        False         False      46h     
image-registry                             4.11.0-rc.0   True        False         False      66m     
ingress                                    4.11.0-rc.0   True        False         False      46h     
kube-apiserver                             4.11.0-rc.0   True        False         False      46h     
kube-controller-manager                    4.11.0-rc.0   True        False         True       46h     GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp: lookup thanos-querier.openshift-monitoring.svc on 10.217.4.10:53: no such host
kube-scheduler                             4.11.0-rc.0   True        False         False      46h     
machine-api                                4.11.0-rc.0   True        False         False      46h     
machine-approver                           4.11.0-rc.0   True        False         False      46h     
machine-config                             4.11.0-rc.0   True        False         False      46h     
marketplace                                4.11.0-rc.0   True        False         False      46h     
network                                    4.11.0-rc.0   True        False         False      46h     
node-tuning                                4.11.0-rc.0   True        False         False      66m     
openshift-apiserver                        4.11.0-rc.0   True        False         False      65m     
openshift-controller-manager               4.11.0-rc.0   True        False         False      67m     
openshift-samples                          4.11.0-rc.0   True        False         False      46h     
operator-lifecycle-manager                 4.11.0-rc.0   True        False         False      46h     
operator-lifecycle-manager-catalog         4.11.0-rc.0   True        False         False      46h     
operator-lifecycle-manager-packageserver   4.11.0-rc.0   True        False         False      71m     
service-ca                                 4.11.0-rc.0   True        False         False      46h     
```
Comment 1 Praveen Kumar 2022-07-05 11:10:46 UTC 
Looking at the commits https://github.com/openshift/cluster-kube-controller-manager-operator/pull/623 is what causes it now. This is now blocking CRC because monitoring stack is disabled by default using the CVO overrides https://github.com/code-ready/snc/blob/master/cvo-overrides.yaml
Comment 3 Adrian 2022-07-06 20:12:05 UTC 
Tested on 4.12.0-0.nightly-2022-07-06-023534 and still got the issue

NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.12.0-0.nightly-2022-07-06-023534   True        False         False      4h52m   
config-operator                            4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
console                                    4.12.0-0.nightly-2022-07-06-023534   True        False         False      4h58m   
dns                                        4.12.0-0.nightly-2022-07-06-023534   True        False         False      5h2m    
etcd                                       4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
image-registry                             4.12.0-0.nightly-2022-07-06-023534   True        False         False      4h57m   
ingress                                    4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
kube-apiserver                             4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
kube-controller-manager                    4.12.0-0.nightly-2022-07-06-023534   True        False         True       34h     GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp: lookup thanos-querier.openshift-monitoring.svc on 10.217.4.10:53: no such host
kube-scheduler                             4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
machine-api                                4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
machine-approver                           4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
machine-config                             4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
marketplace                                4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
network                                    4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
node-tuning                                4.12.0-0.nightly-2022-07-06-023534   True        False         False      4h56m   
openshift-apiserver                        4.12.0-0.nightly-2022-07-06-023534   True        False         False      4h58m   
openshift-controller-manager               4.12.0-0.nightly-2022-07-06-023534   True        False         False      4h57m   
openshift-samples                          4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
operator-lifecycle-manager                 4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
operator-lifecycle-manager-catalog         4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h     
operator-lifecycle-manager-packageserver   4.12.0-0.nightly-2022-07-06-023534   True        False         False      5h2m    
service-ca                                 4.12.0-0.nightly-2022-07-06-023534   True        False         False      34h
Comment 5 Praveen Kumar 2022-07-07 06:53:51 UTC 
I tested the 4.12.0-0.nightly-2022-07-06-221008 and it is now not showing degraded state.

```
$ oc get co
NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.12.0-0.nightly-2022-07-06-221008   True        False         False      6m37s   
config-operator                            4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
console                                    4.12.0-0.nightly-2022-07-06-221008   True        False         False      10m     
dns                                        4.12.0-0.nightly-2022-07-06-221008   True        False         False      15m     
etcd                                       4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
image-registry                             4.12.0-0.nightly-2022-07-06-221008   True        False         False      10m     
ingress                                    4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
kube-apiserver                             4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
kube-controller-manager                    4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
kube-scheduler                             4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
machine-api                                4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
machine-approver                           4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
machine-config                             4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
marketplace                                4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
network                                    4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
node-tuning                                4.12.0-0.nightly-2022-07-06-221008   True        False         False      10m     
openshift-apiserver                        4.12.0-0.nightly-2022-07-06-221008   True        False         False      10m     
openshift-controller-manager               4.12.0-0.nightly-2022-07-06-221008   True        False         False      11m     
openshift-samples                          4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
operator-lifecycle-manager                 4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
operator-lifecycle-manager-catalog         4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h     
operator-lifecycle-manager-packageserver   4.12.0-0.nightly-2022-07-06-221008   True        False         False      16m     
service-ca                                 4.12.0-0.nightly-2022-07-06-221008   True        False         False      26h    
```
Comment 9 errata-xmlrpc 2023-01-17 19:51:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399
Note You need to log in before you can comment on or make changes to this bug.