Description of problem: During cnv upgrade v4.12.4 to 4.12.5-50 (OCP: 4.12.26), noticed KubevirtHyperconvergedClusterOperatorCRModification alert in firing state. Version-Release number of selected component (if applicable): cnv: 4.12.4 to 4.12.5-50 (OCP: 4.12.26) How reproducible: 1 out of 1 attempt Steps to Reproduce: 1. Check alerts fired during cnv upgrade 2. 3. Actual results: {'labels': {'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'role/hyperconverged-cluster-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}, 'annotations': {'description': 'Out-of-band modification for role/hyperconverged-cluster-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}, 'state': 'firing', 'activeAt': '2023-07-26T20:47:55.920372176Z', 'value': '1e+00'}, {'labels': {'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'servicemonitor/kubevirt-hyperconverged-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}, 'annotations': {'description': 'Out-of-band modification for servicemonitor/kubevirt-hyperconverged-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}, 'state': 'firing', 'activeAt': '2023-07-26T20:48:55.920372176Z', 'value': '1e+00'}, {'labels': {'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'rolebinding/hyperconverged-cluster-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}, 'annotations': {'description': 'Out-of-band modification for rolebinding/hyperconverged-cluster-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}, 'state': 'firing', 'activeAt': '2023-07-26T20:48:55.920372176Z', 'value': '1e+00'}, {'labels': {'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'service/kubevirt-hyperconverged-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}, 'annotations': {'description': 'Out-of-band modification for service/kubevirt-hyperconverged-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}, 'state': 'firing', 'activeAt': '2023-07-26T20:48:55.920372176Z', 'value': '1e+00'}] assert not [{'activeAt': '2023-07-26T20:49:22.832759694Z', 'annotations': {'runbook_url': 'https://kubevirt.io/monitoring/runbook...omponent': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', ...}, 'state': 'firing', ...}] Expected results: we are trying to capture all the alerts that are fired during upgrades and reduce the noise generated. Additional info: http://pastebin.test.redhat.com/1106300 must-gather.log http://pastebin.test.redhat.com/1106301
Please try to reproduce
https://bugzilla.redhat.com/show_bug.cgi?id=2154319 this was addressed in 4.13.1. Do we want to backport the fix?
*** Bug 2237487 has been marked as a duplicate of this bug. ***
*** Bug 2224826 has been marked as a duplicate of this bug. ***
This will be fixed by https://github.com/kubevirt/hyperconverged-cluster-operator/pull/2500
QA: Verified Deployed: OCP-4.12.36 Deployed: CNV-v4.12.6-20 >> upgraded to CNV-v4.12.7-19 alertname': 'KubevirtHyperconvergedClusterOperatorCRModification did not fired during the upgrade
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 4.12.7 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:5626