Bug 2012412

Summary: Error provisioning bucket is seen when creating bulk of OBCs
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Tiffany Nguyen <tunguyen>
Component: Multi-Cloud Object GatewayAssignee: Nimrod Becker <nbecker>
Status: CLOSED WORKSFORME QA Contact: Raz Tamir <ratamir>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.9CC: etamir, ocs-bugs, odf-bz-bot
Target Milestone: ---Keywords: Automation, Regression
Target Release: ODF 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-02 15:06:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
obc_pending.log none

Description Tiffany Nguyen 2021-10-09 06:27:09 UTC 
Created attachment 1831100 [details]
obc_pending.log

Created attachment 1831100 [details]
obc_pending.log

Description of problem (please be detailed as possible and provide log
snippests):

After creating a bulk of 400+ OBCs, 1 OBC is stuck at Pending state. 
From noobaa-operator log, there is bucket provisioning error:

E1009 05:13:44.012390       1 controller.go:205] error syncing 'openshift-storage/obc-test-e59034ad2abe4aa3a3ded647f4ba4f9': error provisioning bucket: connection closed ws://[::ffff:10.131.0.37]:58950/(s0wjdpxb) reqid 1578@ws://[::ffff:10.131.0.37]:58950/(s0wjdpxb), requeuing

Bucket at Pending state and never got updated to Bound state:

I1009 05:24:26.333342       1 resourcehandlers.go:446]  "msg"="updating status:" "key"="openshift-storage/obc-test-e59034ad2abe4aa3a3ded647f4ba4f9" "new status"="Pending" "obc"="openshift-storage/obc-test-e59034ad2abe4aa3a3ded647f4ba4f9" "old status"="Pending"


Version of all relevant components (if applicable):
 * noobaa-operator.v4.9.0
 * full_version=4.9.0-182.ci

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Failed scale obc creation test cases.

Is there any workaround available to the best of your knowledge?
None.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
3/3

Can this issue reproduce from the UI?
Not try

If this is a regression, please provide more details to justify this:
Yes

Steps to Reproduce:
1. Creation a bulk of 400+ OBCs 
Test case link: https://github.com/red-hat-storage/ocs-ci/blob/master/tests/e2e/scale/noobaa/test_scale_obc_creation.py
2. Check all OBCs status after creation
3. OBC stuck at Pending state

Actual results:
OBC stuck at Pending state

Expected results:
All OBCs should create and on "Bound" state

Additional info:
Comment 4 Tiffany Nguyen 2021-11-02 15:06:50 UTC 
This issue has not seen with later build 4.9.0-214.ci.