Bug 1850334 - noobaa-default-backing-store-noobaa-pod stuck in Pending State due to PVC issue in OCS 4.5.0-460.ci build
Summary: noobaa-default-backing-store-noobaa-pod stuck in Pending State due to PVC iss...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: OCS 4.5.0
Assignee: Nimrod Becker
QA Contact: Sidhant Agrawal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-24 05:38 UTC by Pratik Surve
Modified: 2023-09-14 06:02 UTC (History)
11 users (show)

Fixed In Version: 4.5.0-463.ci
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-15 10:17:53 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github noobaa noobaa-operator pull 346 0 None closed Fix default storage class missing problem 2021-02-09 01:47:59 UTC
Red Hat Product Errata RHBA-2020:3754 0 None None None 2020-09-15 10:18:21 UTC

Comment 3 Sébastien Han 2020-06-24 08:36:11 UTC
Not sure why this is a Rook issue, which SC is Noobaa consuming the PVC from?

Comment 8 Travis Nielsen 2020-06-24 17:59:29 UTC
The related bug and PV fallback would be related to the OCS operator, so updating the component...

Comment 10 Travis Nielsen 2020-06-24 19:32:26 UTC
I'll defer to Nimrod to confirm on this issue where the PVC is generated incorrectly.

Comment 11 Nimrod Becker 2020-06-25 07:08:53 UTC
fallback is not related to the ocs-op, but in any case, we identified an issue which can cause this. 
I'm moving back to noobaa for now.

Comment 12 Yaniv Kaul 2020-06-25 10:23:09 UTC
devel-ack is missing.

Comment 16 Oded 2020-07-26 11:44:25 UTC
Hi Nimrod,
Do you know how to simulate this bug?

Comment 17 Nimrod Becker 2020-07-27 06:39:23 UTC
RGW was not deployed, that might be the only way to simulate this

Comment 18 Elad 2020-08-02 08:52:20 UTC
From what I see here, in order to verify the fix, we need to try creation of PV pool over BM cluster, using the exact same CRs used by Noobaa upon fallback to PV pool for the case if RGW is not created automatically.

@Nimrod - where can we find these CRs?

Comment 19 Nimrod Becker 2020-08-02 08:53:45 UTC
Use the NooBaa CLI to create a pv pool.

Comment 20 Oded 2020-08-06 14:35:23 UTC
Setup:
Bare Metal
OCP Version:4.5.0-0.nightly-2020-08-03-123303
OCS Version:ocs-operator.v4.5.0-515.ci


Test Process:

1.Create PV-POOL storage class is localblock:
$ nb -n openshift-storage backingstore create pv-pool pv-noobaa-test2 --num-volumes 1 --pv-size-gb 50 --storage-class localblock
ERRO[0603] ❌ BackingStore "pv-noobaa-test2" Phase is "Rejected": Failed connecting all pods in backingstore for more than 10 minutes Current failing: 1 from requested: 1

2.Create PV-POOL storage class is localblock
$ nb -n openshift-storage backingstore create pv-pool pv-noobaa-test --num-volumes 1 --pv-size-gb 50 --storage-class ocs-storagecluster-ceph-rbd
INFO[0117] ✅ BackingStore "pv-noobaa-test" Phase is Ready 

3.Get Pods status:
$ oc get pod -n openshift-storage
NAME                                                              READY   STATUS      RESTARTS   AGE
pv-noobaa-test-noobaa-pod-4915f96b                                1/1     Running     0          4h34m
pv-noobaa-test2-noobaa-pod-74eb5e9b                               0/1     Pending     0          84m

4.Get PVC status:
$ oc get pvc -n openshift-storage
NAME                                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE                  27h
pv-noobaa-test-noobaa-pvc-4915f96b    Bound     pvc-2df985f3-5c47-4c68-b49b-435f998f69f0   50Gi       RWO            ocs-storagecluster-ceph-rbd   4h39m
pv-noobaa-test2-noobaa-pvc-74eb5e9b   Pending                                                                        localblock                    88m

*What is the next step? 
*What are steps to reproduce this bug?

Comment 21 Oded 2020-08-06 14:40:25 UTC
(In reply to Oded from comment #20)
> Setup:
> Bare Metal
> OCP Version:4.5.0-0.nightly-2020-08-03-123303
> OCS Version:ocs-operator.v4.5.0-515.ci
> 
> 
> Test Process:
> 
> 1.Create PV-POOL storage class is localblock:
> $ nb -n openshift-storage backingstore create pv-pool pv-noobaa-test2
> --num-volumes 1 --pv-size-gb 50 --storage-class localblock
> ERRO[0603] ❌ BackingStore "pv-noobaa-test2" Phase is "Rejected": Failed
> connecting all pods in backingstore for more than 10 minutes Current
> failing: 1 from requested: 1
> 
> 2.Create PV-POOL storage class is rbd
> $ nb -n openshift-storage backingstore create pv-pool pv-noobaa-test
> --num-volumes 1 --pv-size-gb 50 --storage-class ocs-storagecluster-ceph-rbd
> INFO[0117] ✅ BackingStore "pv-noobaa-test" Phase is Ready 
> 
> 3.Get Pods status:
> $ oc get pod -n openshift-storage
> NAME                                                              READY  
> STATUS      RESTARTS   AGE
> pv-noobaa-test-noobaa-pod-4915f96b                                1/1    
> Running     0          4h34m
> pv-noobaa-test2-noobaa-pod-74eb5e9b                               0/1    
> Pending     0          84m
> 
> 4.Get PVC status:
> $ oc get pvc -n openshift-storage
> NAME                                  STATUS    VOLUME                      
> CAPACITY   ACCESS MODES   STORAGECLASS                  AGE                 
> 27h
> pv-noobaa-test-noobaa-pvc-4915f96b    Bound    
> pvc-2df985f3-5c47-4c68-b49b-435f998f69f0   50Gi       RWO           
> ocs-storagecluster-ceph-rbd   4h39m
> pv-noobaa-test2-noobaa-pvc-74eb5e9b   Pending                               
> localblock                    88m
> 
> *What is the next step? 
> *What are steps to reproduce this bug?

Comment 22 Harish NV Rao 2020-08-06 14:43:39 UTC
@Pratik, can you please check?

Comment 23 Harish NV Rao 2020-08-06 14:44:17 UTC
@Pratik, can you please check?

Comment 26 errata-xmlrpc 2020-09-15 10:17:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Container Storage 4.5.0 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3754

Comment 27 Red Hat Bugzilla 2023-09-14 06:02:44 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.