Bug 1968180

Summary: noobaa-db-pg-0 in status Terminating
Product: [Red Hat Storage] Red Hat OpenShift Container Storage Reporter: Aviad Polak <apolak>
Component: Multi-Cloud Object GatewayAssignee: Nimrod Becker <nbecker>
Status: CLOSED NOTABUG QA Contact: Raz Tamir <ratamir>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.8CC: ebenahar, etamir, ocs-bugs
Target Milestone: ---Keywords: AutomationBackLog
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-07 16:04:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
noobaa diagnostic none

Description Aviad Polak 2021-06-06 14:30:58 UTC 
Created attachment 1789119 [details]
noobaa diagnostic

Description of problem:
cluster is running for a while, there is 2 osds down, and noobaa-db-pg-0 pod is in "Terminating" status. unable to delete this pod


Version-Release number of selected component (if applicable):
ocs-operator.v4.8.0-402.ci
Server Version: 4.8.0-0.nightly-2021-05-25-041803
Kubernetes Version: v1.21.0-rc.0+ee60d07


How reproducible:
not sure how to reproduce. cluster is running for a while, and some steps in order to setup rbd-rbd mirroring done already. (here: https://red-hat-storage.github.io/ocs-training/RegionalDR/manual/ocs4-multisite-replication.html#_configuring_rbd_mirroring_between_odf_clusters 1.2, 1.3)

nodes status:
$ oc get nodes
NAME              STATUS   ROLES    AGE   VERSION
compute-0         Ready    worker   12d   v1.21.0-rc.0+ee60d07
compute-1         Ready    worker   12d   v1.21.0-rc.0+ee60d07
compute-2         Ready    worker   12d   v1.21.0-rc.0+ee60d07
control-plane-0   Ready    master   12d   v1.21.0-rc.0+ee60d07
control-plane-1   Ready    master   12d   v1.21.0-rc.0+ee60d07
control-plane-2   Ready    master   12d   v1.21.0-rc.0+ee60d07


Additional info:
attaching: ocs must-gather, noobaa diagnostic

must-gather: https://drive.google.com/file/d/1U33V3xQvM9ZQ9c9bwRM2CkOOMQG4SwPW/view?usp=sharing
Comment 3 Aviad Polak 2021-06-07 13:14:04 UTC 
root cause is here:

https://bugzilla.redhat.com/show_bug.cgi?id=1434709 
https://github.com/kubernetes/kubernetes/issues/45679#issuecomment-301365878

this bz can be closed