Description of problem (please be detailed as possible and provide log snippests): Alerts CephMgrIsAbsent and CephMgrIsMissingReplicas are not triggered when deployment rook-ceph-mgr-a has no pods. Version of all relevant components (if applicable): OCS 4.8.3-10 OCP 4.8.0-0.nightly-2021-10-09-130601 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Ceph alerts are not working when Ceph Manager is not working. This can hide any Ceph problem in the cluster by not informing user. Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? yes Can this issue reproduce from the UI? yes If this is a regression, please provide more details to justify this: The alert worked in previous versions. Steps to Reproduce: 1. Downscale rook-ceph-mgr-a deployment (Workloads -> Deployments -> rook-ceph-mgr-a in OCP Console). 2. Check alerts (Monitoring -> Alerting in OCP Console). Actual results: It seems that alerts CephMgrIsAbsent and CephMgrIsMissingReplicas are raised for few seconds but are immediately resolved even if the problem is not resolved yet. Those alerts are not triggered anymore. Expected results: Alerts CephMgrIsAbsent and CephMgrIsMissingReplicas should work. Additional info:
We **MUST** always open a new bug against ODF and current release (which is 4.9 currently) It does not matter where the bug was found, it will be decided that where the fix will go but we must always start with the current release.
Added the following PR: https://github.com/rook/rook/pull/8985 Sebastian, Travis please take a look
Adding RAC (same as commit message) CephMgrIsAbsent ---------------- This alert initially had the following query absent(up{job="rook-ceph-mgr"}) which will fire when the 'up' query is not present, but had two flows a. it will not be fired if 'up' provides a result with ZERO value b. it will not give any fields in the metric, so 'namespace' was missing when the above query was replaced with the following, up{job="rook-ceph-mgr"} == 0 query had the following shortage a. whenever mgr pod is completely down (like 'replicas' set to ZERO and 'mgr' is not coming up), 'up' query will not give any result. Thus we had to combine both the queries to get results in both the scenarios. CephMgrIsMissingReplicas ------------------------ This query previously was, sum(up{job="rook-ceph-mgr"}) < 1 had the same structure as the above (Absent) query, but it's intention was to check the no: of 'replicas' count for ceph mgr. Now it is changed to a kube query which handles the replicas count.
The fix looks ok. -> VERIFIED Tested with: OCS 4.8.3-14 OCP 4.8.0-0.nightly-2021-10-16-024756
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OCS 4.8.3 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3881