Bug 2232356

Summary:	[MCG]Noobaa OBC stuck pending NO_SUCH_BUCKET
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	khover
Component:	Multi-Cloud Object Gateway	Assignee:	Nimrod Becker <nbecker>
Status:	NEW ---	QA Contact:	krishnaram Karthick <kramdoss>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.12	CC:	azaky, dzaken, nbecker, odf-bz-bot
Target Milestone:	---	Flags:	khover: needinfo? (nbecker)
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description khover 2023-08-16 13:33:56 UTC

Description of problem (please be detailed as possible and provide log
snippests):

ODF 4.12 fresh install

Noobaa OBC stuck pending NO_SUCH_BUCKET

Version of all relevant components (if applicable):

ODF 4.12.5
mcg-operator.v4.12.5-rhodf

OVN network 

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Customer expects OBC to not be stuck pending on initial install.

Is there any workaround available to the best of your knowledge?

Only restarting all Noobaa pods moves obc from Pending to Bound

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:

Comment 2 khover 2023-08-16 13:39:02 UTC

operator and core logs prior to restarting noobaa stack 

time="2023-08-10T09:55:55Z" level=info msg="SetPhase: \"Ready\"" sys=openshift-storage/noobaa
time="2023-08-10T09:55:55Z" level=info msg="✅ Done" sys=openshift-storage/noobaa
time="2023-08-10T09:55:55Z" level=info msg="UpdateStatus: Done generation 3" sys=openshift-storage/noobaa
I0810 09:56:00.678385       1 controller.go:218]  "msg"="reconciling claim" "key"="default/test1" 
I0810 09:56:00.678401       1 helpers.go:107]  "msg"="getting claim for key" "key"="default/test1" 
I0810 09:56:00.681324       1 helpers.go:213]  "msg"="getting ObjectBucketClaim's StorageClass" "key"="default/test1" 
I0810 09:56:00.684189       1 helpers.go:218]  "msg"="got StorageClass" "key"="default/test1" "name"="openshift-storage.noobaa.io"
I0810 09:56:00.684248       1 helpers.go:90]  "msg"="checking OBC for OB name, this indicates provisioning is complete" "key"="default/test1" "test1"=null
I0810 09:56:00.684291       1 resourcehandlers.go:446]  "msg"="updating status:" "key"="default/test1" "new status"="Pending" "obc"="default/test1" "old status"=""
I0810 09:56:00.689904       1 controller.go:283]  "msg"="syncing obc creation" "key"="default/test1" 
I0810 09:56:00.689953       1 controller.go:630]  "msg"="updating OBC metadata" "key"="default/test1" 
I0810 09:56:00.689976       1 resourcehandlers.go:436]  "msg"="updating" "key"="default/test1" "obc"="default/test1"
I0810 09:56:00.695978       1 resourcehandlers.go:436]  "msg"="updating" "key"="default/test1" "obc"="default/test1"
E0810 09:56:00.701239       1 controller.go:669]  "msg"="invalid changes to OBC. only additionalConfig can be updated" "error"=null "key"="default/test1" 
I0810 09:56:00.701282       1 resourcehandlers.go:148]  "msg"="seeing if OB for OBC exists" "key"="default/test1" "checking for OB name"="obc-default-test1"
I0810 09:56:00.704407       1 controller.go:406]  "msg"="provisioning" "key"="default/test1" "bucket"="test1-13646b49-13bd-4401-a073-925b5ee0f396"
time="2023-08-10T09:56:00Z" level=info msg="Provision: got request to provision bucket \"test1-13646b49-13bd-4401-a073-925b5ee0f396\"" provisioner=openshift-storage.noobaa.io/obc
time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: Service \"noobaa-mgmt\"\n"
time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: Secret \"noobaa-operator\"\n"
time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: Secret \"noobaa-admin\"\n"
time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: BucketClass \"noobaa-default-bucket-class\"\n"
time="2023-08-10T09:56:00Z" level=info msg="✈️  RPC: bucket.read_bucket() Request: {Name:test1-13646b49-13bd-4401-a073-925b5ee0f396}"
time="2023-08-10T09:56:00Z" level=error msg="⚠️  RPC: bucket.read_bucket() Response Error: Code=NO_SUCH_BUCKET Message=No such bucket: test1-13646b49-13bd-4401-a073-925b5ee0f396"
time="2023-08-10T09:56:00Z" level=info msg="Provisioner: replication policy " provisioner=openshift-storage.noobaa.io/obc
time="2023-08-10T09:56:00Z" level=info msg="✈️  RPC: tier.create_tier() Request: {Name:test1-13646b49-13bd-4401-a073-925b5ee0f396.64d4b430.0 DataPlacement:SPREAD AttachedPools:[noobaa-default-backing-store] ChunkCoderConfig
:<nil>}"

[32mAug-10 9:55:55.377[35m [BGWorkers/37] [36m   [L0][39m core.server.bg_services.agent_blocks_reclaimer:: AGENT_BLOCKS_RECLAIMER: BEGIN
[32mAug-10 9:56:00.726[35m [WebServer/41] [31m[ERROR][39m core.server.system_services.bucket_server:: BUCKET NOT FOUND SENSITIVE-0bf5499322b60df3
[32mAug-10 9:56:00.726[35m [WebServer/41] [31m[ERROR][39m CONSOLE:: RPC._on_request: ERROR srv bucket_api.read_bucket reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-233 connid ws://[::ffff:10.128.9.39]:54920/(4reyljgr) Error: No such bucket: test1-13646b49-13bd-4401-a073-925b5ee0f396
    at find_bucket (/root/node_modules/noobaa-core/src/server/system_services/bucket_server.js:1294:15)
    at read_bucket_sdk_info (/root/node_modules/noobaa-core/src/server/system_services/bucket_server.js:467:20)
    at Object.read_bucket (/root/node_modules/noobaa-core/src/server/system_services/bucket_server.js:462:35)
    at Object.server_func (/root/node_modules/noobaa-core/src/rpc/rpc.js:113:48)
    at /root/node_modules/noobaa-core/src/rpc/rpc.js:340:46
    at RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:345:19)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5


All Logs captured so far:

supportshell /cases/03583861

drwxrwxrwx+ 3 yank yank    4096 Aug 10 12:20 0090-odf-mustgather-before-08102023.tar.gz
-rw-rw-rw-+ 1 yank yank   76617 Aug 10 12:20 0100-rook-ceph-operator-pod.log
-rw-rw-rw-+ 1 yank yank  600190 Aug 10 12:20 0110-noobaa-core-0-pod.log
-rw-rw-rw-+ 1 yank yank   13824 Aug 10 12:20 0120-before-collected-logs.txt
-rw-rw-rw-+ 1 yank yank 1060086 Aug 10 12:20 0130-noobaa-operator-pod.log
drwxrwxrwx+ 3 yank yank    4096 Aug 10 12:25 0140-odf-mustgather-after-08102023.tar.gz
-rw-rw-rw-+ 1 yank yank 5304002 Aug 10 12:25 0150-noobaa-operator.log
-rw-rw-rw-+ 1 yank yank  192344 Aug 10 12:25 0160-rook-ceph-operator-pod.log
-rw-rw-rw-+ 1 yank yank  625848 Aug 10 12:25 0170-after-collected-logs.txt
-rw-rw-rw-+ 1 yank yank 1190257 Aug 10 12:25 0180-noobaa-core-0-pod.log