Bug 1964958

Summary:	StretchCluster \| Arbiter \| node become Not_Ready when running IO
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	Avi Liani <alayani>
Component:	ceph	Assignee:	Scott Ostapovicz <sostapov>
Status:	CLOSED DUPLICATE	QA Contact:	Raz Tamir <ratamir>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.7	CC:	aclewett, bniver, ebenahar, madam, mbukatov, muagarwa, nberry, ocs-bugs, tnielsen
Target Milestone:	---	Keywords:	AutomationBackLog, TestBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-06-14 10:16:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 3 Mudit Agarwal 2021-05-26 13:17:05 UTC

Travis, PTAL

Comment 4 Travis Nielsen 2021-05-26 15:30:11 UTC

How was the latency injected? I assume it affects the entire OCP cluster, including etcd. But this really is a question for OCP since the nodes could not tolerate the longer latency. The node status is not directly affected by OCS.

Comment 5 Travis Nielsen 2021-06-02 18:45:05 UTC

Avi Can you add more details about the latency injection before we raise this with OCP?

Comment 7 Avi Liani 2021-06-03 09:08:04 UTC

I hit the same issue when i set the latency to : 0

Comment 9 Travis Nielsen 2021-06-03 18:12:22 UTC

Avi Can you test again with at least 12 vCPUs as suggested by Annette?

Comment 10 Avi Liani 2021-06-07 05:56:39 UTC

I ran the test again with 16 vCPUs for each worker, and i end up with the same behaviour, 2 workers become into NotReady state and the CPU consumption reach 100%.

Comment 13 Mudit Agarwal 2021-06-10 06:35:44 UTC

Hi Avi, any update on this?

Comment 14 Avi Liani 2021-06-10 06:51:40 UTC

Not yet.
Hopefully I will get the hardware on Mondaye (Jun-14) and i will rerun the tests then.

Comment 15 Martin Bukatovic 2021-06-11 18:37:17 UTC

I wonder what is consuming the cpu exactly.

Could you query a cpu utilization in Prometheus or Grafana per namespace/project? And from there, narrow it down to individual pods?

Comment 17 Mudit Agarwal 2021-06-14 10:16:38 UTC

Duping it to BZ #1945016 after discussing with Elad, we have seen this issue many times now where the worker nodes go down while running FIO.
Please update the master bug with the new findings.

*** This bug has been marked as a duplicate of bug 1945016 ***