Bug 1964958 - StretchCluster | Arbiter | node become Not_Ready when running IO
Summary: StretchCluster | Arbiter | node become Not_Ready when running IO
Keywords:
Status: CLOSED DUPLICATE of bug 1945016
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: ceph
Version: 4.7
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Scott Ostapovicz
QA Contact: Raz Tamir
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-26 12:30 UTC by Avi Liani
Modified: 2021-06-14 10:16 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-14 10:16:38 UTC
Embargoed:

Attachments (Terms of Use)

Comment 3 Mudit Agarwal 2021-05-26 13:17:05 UTC 
Travis, PTAL
Comment 4 Travis Nielsen 2021-05-26 15:30:11 UTC 
How was the latency injected? I assume it affects the entire OCP cluster, including etcd. But this really is a question for OCP since the nodes could not tolerate the longer latency. The node status is not directly affected by OCS.
Comment 5 Travis Nielsen 2021-06-02 18:45:05 UTC 
Avi Can you add more details about the latency injection before we raise this with OCP?
Comment 7 Avi Liani 2021-06-03 09:08:04 UTC 
I hit the same issue when i set the latency to : 0
Comment 9 Travis Nielsen 2021-06-03 18:12:22 UTC 
Avi Can you test again with at least 12 vCPUs as suggested by Annette?
Comment 10 Avi Liani 2021-06-07 05:56:39 UTC 
I ran the test again with 16 vCPUs for each worker, and i end up with the same behaviour, 2 workers become into NotReady state and the CPU consumption reach 100%.
Comment 13 Mudit Agarwal 2021-06-10 06:35:44 UTC 
Hi Avi, any update on this?
Comment 14 Avi Liani 2021-06-10 06:51:40 UTC 
Not yet.
Hopefully I will get the hardware on Mondaye (Jun-14) and i will rerun the tests then.
Comment 15 Martin Bukatovic 2021-06-11 18:37:17 UTC 
I wonder what is consuming the cpu exactly.

Could you query a cpu utilization in Prometheus or Grafana per namespace/project? And from there, narrow it down to individual pods?
Comment 17 Mudit Agarwal 2021-06-14 10:16:38 UTC 
Duping it to BZ #1945016 after discussing with Elad, we have seen this issue many times now where the worker nodes go down while running FIO.
Please update the master bug with the new findings.

*** This bug has been marked as a duplicate of bug 1945016 ***
Note You need to log in before you can comment on or make changes to this bug.