Bug 2051661

Summary: [GSS] Unable to Access Object Buckets [Noobaa BackingStore In Rejected Phase]
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Craig Wayman <crwayman>
Component: cephAssignee: Matt Benjamin (redhat) <mbenjamin>
ceph sub component: RBD QA Contact: Neha Berry <nberry>
Status: CLOSED WORKSFORME Docs Contact:
Severity: high    
Priority: unspecified CC: bkunal, bniver, etamir, hnallurv, jquinn, jthottan, lmauda, madam, mbenjamin, mhackett, muagarwa, nbecker, ocs-bugs, odf-bz-bot, pnataraj
Version: 4.7   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-19 06:04:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Errors none

Description Craig Wayman 2022-02-07 18:00:18 UTC 
Created attachment 1859622 [details]
Errors

Description of problem (please be detailed as possible and provide log
snippets): 

  The customer began experiencing MCG issues and was seeing the error of "Backing store mode: INITIALIZING" after an incremental update within OCS v4.6.

  The customer was informed to upgrade to OCS v4.7 which should've fixed the issue (referenced Bug 1911266). Currently, the customer is viewing numerous errors and is unable to access previously established buckets. The following  events/errors are displayed:

NOT ENOUGH HEALTH RESOURCES

Buckets are in Error States

Service is Degraded

Backing store mode: ALL_NODES_OFFLINE

Warning  BackingStorePhaseRejected


InvalidBucketState: The request is not valid with the current state of the bucket. status code: 409

Version of all relevant components (if applicable):
OCS 4.6 and after upgrade to OCS 4.7 issue still persists.

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Yes, the customer does not have access to data in previously established buckets. 

Is there any workaround available to the best of your knowledge?

No, Tried restoring Multicloud-Object-Gateway IAW Chapter 10 (4.7 OCS Troubleshooting Product Doc). i.e. Restarted all the pods related to the MCG and restarted the RADOS Object Gateway (RGW) pod.

After performing the steps to restore MCG, the customer is now able to create buckets but still cannot access the previous buckets.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?



Can this issue reproducible?

No


Can this issue reproduce from the UI?

No


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:
Comment 3 Jiffin 2022-03-31 05:44:46 UTC 
can you please attach the must gather or logs to see the status of rgw pods and service. There is very limited info for debugging this issue
Comment 12 Craig Wayman 2022-06-20 14:10:16 UTC 
After being provided with the information in case comment #85, the customer has closed the case. The issue seems to be resolved, we can close this BZ. 

Regards,


Craig Wayman
TSE Red Hat OpenShift Data Foundations (ODF) 
Customer Experience and Engagement, NA