Hide Forgot
Description of problem: In KAS-O logs, below event is high-frequency looped: 2021-05-06T07:09:03.614448160Z I0506 07:09:03.614391 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"54086850-4a33-4773-96c1-6828b83ba46e", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PrometheusRuleUpdated' Updated PrometheusRule.monitoring.coreos.com/v1 because it changed ...<snipped another 12 same events before next second 2021-05-06T07:10>... Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-05-06-003426 How reproducible: Always Steps to Reproduce: 1. Check KAS-O logs Actual results: 1. Above log is high-frequency looped. $ grep 'Updated PrometheusRule.monitoring.coreos.com/v1 because it changed' kas-o.log | wc -l 3033 Then changed operatorLogLevel to Debug: $ oc edit kubeapiserver cluster ... operatorLogLevel: Debug ... Checked KAS-O logs again, found: 2021-05-06T07:48:03.481235292Z I0506 07:48:03.481184 1 monitoring.go:122] PrometheusRule "openshift-kube-apiserver/cpu-utilization" changes: {} 2021-05-06T07:48:03.490663221Z I0506 07:48:03.490596 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"54086850-4a33-4773-96c1-6828b83ba46e", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PrometheusRuleUpdated' Updated PrometheusRule.monitoring.coreos.com/v1 because it changed Each event is preceded by `"openshift-kube-apiserver/cpu-utilization" changes: {}`. That is, actual change is always empty {}, i.e. actually no change at all. Double checked `oc get PrometheusRule cpu-utilization -n openshift-kube-apiserver --watch`, the output shows one item, then never has new output, i.e. actually no change either. Repeating `oc get PrometheusRule cpu-utilization -o yaml`, the output never "changes" either between repeated executions. Expected results: 1. No loop, given cpu-utilization actually has no any "changes". Additional info: Related bug: bug 1951705. Seems introduced by that bug.
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-05-26-172234 True False 167m Cluster version is 4.8.0-0.nightly-2021-05-26-172234 $ oc get po -n openshift-kube-apiserver-operator NAME READY STATUS RESTARTS AGE kube-apiserver-operator-5cd5cd7fc5-r9cs5 1/1 Running 2 3h5m Checked the kube-apiserver-operator log file, $ oc logs -n openshift-kube-apiserver-operator kube-apiserver-operator-5cd5cd7fc5-r9cs5 | grep 'Updated PrometheusRule.monitoring.coreos.com/v1 because it changed' No long seen the message, so move the bug VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438