Bug 2242132

Summary: CephNodeDown alert does not appear
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Daniel Osypenko <dosypenk>
Component: ceph-monitoringAssignee: Juan Miguel Olmo <jolmomar>
Status: CLOSED DUPLICATE QA Contact: Bipul Adhikari <badhikar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.14CC: jolmomar, odf-bz-bot
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-10-09 09:43:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Osypenko 2023-10-04 14:42:52 UTC 
Description of problem (please be detailed as possible and provide log
snippests):

When powering off a worker node abruptly or gracefully via vCenter or terraform a CephNodeDown does not appear OCP 4.14 ODF 4.14. Cluster was working without node for 20 min.

% oc get nodes
NAME              STATUS     ROLES                  AGE    VERSION
compute-0         Ready      worker                 2d4h   v1.27.6+6936c15
compute-1         NotReady   worker                 2d4h   v1.27.6+6936c15
compute-2         Ready      worker                 2d4h   v1.27.6+6936c15
control-plane-0   Ready      control-plane,master   2d4h   v1.27.6+6936c15
control-plane-1   Ready      control-plane,master   2d4h   v1.27.6+6936c15
control-plane-2   Ready      control-plane,master   2d4h   v1.27.6+6936c15


Version of all relevant components (if applicable):
OC version:
Client Version: 4.13.4
Kustomize Version: v4.5.7
Server Version: 4.14.0-0.nightly-2023-09-29-231104
Kubernetes Version: v1.27.6+6936c15

OCS verison:
ocs-operator.v4.14.0-139.stable              OpenShift Container Storage   4.14.0-139.stable              Succeeded

Cluster version
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.0-0.nightly-2023-09-29-231104   True        False         2d5h    Cluster version is 4.14.0-0.nightly-2023-09-29-231104

Rook version:
rook: v4.14.0-0.94cb34310a94b9e21117038c1b3b416d01676b36
go: go1.20.5

Ceph version:
ceph version 17.2.6-146.el9cp (1d01c2b30b5fd39787bb8804707c4b2e52e30137) quincy (stable)


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
no

Is there any workaround available to the best of your knowledge?
no

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
yes, every time

Can this issue reproduce from the UI?
yes

If this is a regression, please provide more details to justify this:
CephNodeDown alert was tested before and it was working good on ODF 4.13 (will update with result for ODF 4.13)

Steps to Reproduce:
1. Turn of any worker node and check appearing alerts via management-console or curl request to prometheus endpoint
2.
3.


Actual results:
CephNodeDown is not triggered when worker node is down

Expected results:
CephNodeDown is triggered when worker node is down

Additional info:
must-gather logs: https://url.corp.redhat.com/a70080a