Bug 1910705 - OBCs are stuck in a Pending state
Summary: OBCs are stuck in a Pending state
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.7
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: OCS 4.7.0
Assignee: Nimrod Becker
QA Contact: aberner
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-24 11:01 UTC by Ben Eli
Modified: 2021-06-01 08:48 UTC (History)
8 users (show)

Fixed In Version: v4.7.0-228.ci
Doc Type: Bug Fix
Doc Text:
Cause: The Custom Resource Definition for Object Bucket Claims (OBCs) was missing a parameter in Rook, causing the bound state to be lost and OBCs to be reconciled repeatedly in a loop. Consequence: Object Bucket Claims (OBCs) would move to the "Bound" state but revert back to "Pending" soon after. Fix: Update CRDs for Rook and add some minor behavioral changes to the OBC library to support recovering from the degraded state. Result: Legacy OBCs that have been successfully created but are in "Pending" phase can be automatically recovered and recognized as "Bound", and new OBCs will reach the "Bound" state properly.
Clone Of:
Environment:
Last Closed: 2021-05-19 09:17:08 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github noobaa noobaa-operator pull 507 0 None closed added objectBucketName property to OBC CRD 2021-02-17 13:34:21 UTC
Github noobaa noobaa-operator pull 508 0 None closed backport to 5.7: added objectBucketName property to OBC CRD 2021-02-17 13:34:22 UTC
Red Hat Product Errata RHSA-2021:2041 0 None None None 2021-05-19 09:17:59 UTC

Description Ben Eli 2020-12-24 11:01:06 UTC
Description of problem (please be detailed as possible and provide log
snippests):
Whenever I try to create OBCs on 4.7, they're stuck in a Pending state. Their description doesn't contain any logs.
When I try to delete them, the deletion hangs - even when using --force and --grace-period 0 together.

Version of all relevant components (if applicable):
OCP 4.7.0-0.nightly-2020-12-20-055006
OCS 4.7.0-200.ci

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes. I cannot create/use OBCs.

Is there any workaround available to the best of your knowledge?
No.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
It is - OBCs are working fine in 4.6.

Steps to Reproduce:
1. Create an OBC using a YAML or the NooBaa CLI

Actual results:
OBC is stuck in a Pending state

Expected results:
OBC should be in a Bound state

Additional info:

Comment 2 Danny 2020-12-28 09:40:47 UTC
After updating noobaa-operator to use the latest lib-bucket-provisioner from master the behavior changed a bit but still buggy - the OBC now bounces between Bound and Pending.

This bug also affects rook provisioner and it's not specific to noobaa

Comment 3 Blaine Gardner 2021-01-04 17:55:23 UTC
After chatting with Danny for a bit in my morning, I am pretty sure the original issue is fixed by the new lib-bucket-provisioner changes I made, but there is an additional issue that the `objectBucketName` field needs to be added to the objectbucketclaim CRD for NooBaa.

The field was not added to CRDs in lib-bucket-provisioner when the feature was added 1 to 2 years ago, but it does exist in Rook. I have gathered from Danny that the lib-bucket-provisioner CRDs for NooBaa are maintained with OCS. It may be a good idea to revisit where the objectbucket and objectbucketclaim CRDs are versioned/maintained as a workflow for both NooBaa and Rook until replaced by COSI [1].

[1] https://github.com/kubernetes-sigs/container-object-storage-interface-controller

Comment 6 aberner 2021-01-14 09:05:02 UTC
Created a new obc as described in the bug and it got to state Bound as desired 

OCP:4.7.0-0.nightly-2021-01-09-231517
OCS:4.7.0-228.ci

Comment 12 errata-xmlrpc 2021-05-19 09:17:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Container Storage 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2041


Note You need to log in before you can comment on or make changes to this bug.