Bug 1957645
| Summary: | Event "Updated PrometheusRule.monitoring.coreos.com/v1 because it changed" is frequently looped with weird empty {} changes | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Xingxing Xia <xxia> |
| Component: | kube-apiserver | Assignee: | David Eads <deads> |
| Status: | CLOSED ERRATA | QA Contact: | Ke Wang <kewang> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.8 | CC: | aos-bugs, mfojtik, xxia |
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 23:06:39 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-05-26-172234 True False 167m Cluster version is 4.8.0-0.nightly-2021-05-26-172234 $ oc get po -n openshift-kube-apiserver-operator NAME READY STATUS RESTARTS AGE kube-apiserver-operator-5cd5cd7fc5-r9cs5 1/1 Running 2 3h5m Checked the kube-apiserver-operator log file, $ oc logs -n openshift-kube-apiserver-operator kube-apiserver-operator-5cd5cd7fc5-r9cs5 | grep 'Updated PrometheusRule.monitoring.coreos.com/v1 because it changed' No long seen the message, so move the bug VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Description of problem: In KAS-O logs, below event is high-frequency looped: 2021-05-06T07:09:03.614448160Z I0506 07:09:03.614391 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"54086850-4a33-4773-96c1-6828b83ba46e", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PrometheusRuleUpdated' Updated PrometheusRule.monitoring.coreos.com/v1 because it changed ...<snipped another 12 same events before next second 2021-05-06T07:10>... Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-05-06-003426 How reproducible: Always Steps to Reproduce: 1. Check KAS-O logs Actual results: 1. Above log is high-frequency looped. $ grep 'Updated PrometheusRule.monitoring.coreos.com/v1 because it changed' kas-o.log | wc -l 3033 Then changed operatorLogLevel to Debug: $ oc edit kubeapiserver cluster ... operatorLogLevel: Debug ... Checked KAS-O logs again, found: 2021-05-06T07:48:03.481235292Z I0506 07:48:03.481184 1 monitoring.go:122] PrometheusRule "openshift-kube-apiserver/cpu-utilization" changes: {} 2021-05-06T07:48:03.490663221Z I0506 07:48:03.490596 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"54086850-4a33-4773-96c1-6828b83ba46e", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PrometheusRuleUpdated' Updated PrometheusRule.monitoring.coreos.com/v1 because it changed Each event is preceded by `"openshift-kube-apiserver/cpu-utilization" changes: {}`. That is, actual change is always empty {}, i.e. actually no change at all. Double checked `oc get PrometheusRule cpu-utilization -n openshift-kube-apiserver --watch`, the output shows one item, then never has new output, i.e. actually no change either. Repeating `oc get PrometheusRule cpu-utilization -o yaml`, the output never "changes" either between repeated executions. Expected results: 1. No loop, given cpu-utilization actually has no any "changes". Additional info: Related bug: bug 1951705. Seems introduced by that bug.