Description of problem (please be detailed as possible and provide log snippests): ODF 4.12 fresh install Noobaa OBC stuck pending NO_SUCH_BUCKET Version of all relevant components (if applicable): ODF 4.12.5 mcg-operator.v4.12.5-rhodf OVN network Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Customer expects OBC to not be stuck pending on initial install. Is there any workaround available to the best of your knowledge? Only restarting all Noobaa pods moves obc from Pending to Bound Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
operator and core logs prior to restarting noobaa stack time="2023-08-10T09:55:55Z" level=info msg="SetPhase: \"Ready\"" sys=openshift-storage/noobaa time="2023-08-10T09:55:55Z" level=info msg="✅ Done" sys=openshift-storage/noobaa time="2023-08-10T09:55:55Z" level=info msg="UpdateStatus: Done generation 3" sys=openshift-storage/noobaa I0810 09:56:00.678385 1 controller.go:218] "msg"="reconciling claim" "key"="default/test1" I0810 09:56:00.678401 1 helpers.go:107] "msg"="getting claim for key" "key"="default/test1" I0810 09:56:00.681324 1 helpers.go:213] "msg"="getting ObjectBucketClaim's StorageClass" "key"="default/test1" I0810 09:56:00.684189 1 helpers.go:218] "msg"="got StorageClass" "key"="default/test1" "name"="openshift-storage.noobaa.io" I0810 09:56:00.684248 1 helpers.go:90] "msg"="checking OBC for OB name, this indicates provisioning is complete" "key"="default/test1" "test1"=null I0810 09:56:00.684291 1 resourcehandlers.go:446] "msg"="updating status:" "key"="default/test1" "new status"="Pending" "obc"="default/test1" "old status"="" I0810 09:56:00.689904 1 controller.go:283] "msg"="syncing obc creation" "key"="default/test1" I0810 09:56:00.689953 1 controller.go:630] "msg"="updating OBC metadata" "key"="default/test1" I0810 09:56:00.689976 1 resourcehandlers.go:436] "msg"="updating" "key"="default/test1" "obc"="default/test1" I0810 09:56:00.695978 1 resourcehandlers.go:436] "msg"="updating" "key"="default/test1" "obc"="default/test1" E0810 09:56:00.701239 1 controller.go:669] "msg"="invalid changes to OBC. only additionalConfig can be updated" "error"=null "key"="default/test1" I0810 09:56:00.701282 1 resourcehandlers.go:148] "msg"="seeing if OB for OBC exists" "key"="default/test1" "checking for OB name"="obc-default-test1" I0810 09:56:00.704407 1 controller.go:406] "msg"="provisioning" "key"="default/test1" "bucket"="test1-13646b49-13bd-4401-a073-925b5ee0f396" time="2023-08-10T09:56:00Z" level=info msg="Provision: got request to provision bucket \"test1-13646b49-13bd-4401-a073-925b5ee0f396\"" provisioner=openshift-storage.noobaa.io/obc time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n" time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: Service \"noobaa-mgmt\"\n" time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: Secret \"noobaa-operator\"\n" time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: Secret \"noobaa-admin\"\n" time="2023-08-10T09:56:00Z" level=info msg="✅ Exists: BucketClass \"noobaa-default-bucket-class\"\n" time="2023-08-10T09:56:00Z" level=info msg="✈️ RPC: bucket.read_bucket() Request: {Name:test1-13646b49-13bd-4401-a073-925b5ee0f396}" time="2023-08-10T09:56:00Z" level=error msg="⚠️ RPC: bucket.read_bucket() Response Error: Code=NO_SUCH_BUCKET Message=No such bucket: test1-13646b49-13bd-4401-a073-925b5ee0f396" time="2023-08-10T09:56:00Z" level=info msg="Provisioner: replication policy " provisioner=openshift-storage.noobaa.io/obc time="2023-08-10T09:56:00Z" level=info msg="✈️ RPC: tier.create_tier() Request: {Name:test1-13646b49-13bd-4401-a073-925b5ee0f396.64d4b430.0 DataPlacement:SPREAD AttachedPools:[noobaa-default-backing-store] ChunkCoderConfig :<nil>}" [32mAug-10 9:55:55.377[35m [BGWorkers/37] [36m [L0][39m core.server.bg_services.agent_blocks_reclaimer:: AGENT_BLOCKS_RECLAIMER: BEGIN [32mAug-10 9:56:00.726[35m [WebServer/41] [31m[ERROR][39m core.server.system_services.bucket_server:: BUCKET NOT FOUND SENSITIVE-0bf5499322b60df3 [32mAug-10 9:56:00.726[35m [WebServer/41] [31m[ERROR][39m CONSOLE:: RPC._on_request: ERROR srv bucket_api.read_bucket reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-233 connid ws://[::ffff:10.128.9.39]:54920/(4reyljgr) Error: No such bucket: test1-13646b49-13bd-4401-a073-925b5ee0f396 at find_bucket (/root/node_modules/noobaa-core/src/server/system_services/bucket_server.js:1294:15) at read_bucket_sdk_info (/root/node_modules/noobaa-core/src/server/system_services/bucket_server.js:467:20) at Object.read_bucket (/root/node_modules/noobaa-core/src/server/system_services/bucket_server.js:462:35) at Object.server_func (/root/node_modules/noobaa-core/src/rpc/rpc.js:113:48) at /root/node_modules/noobaa-core/src/rpc/rpc.js:340:46 at RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:345:19) at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:95:5 All Logs captured so far: supportshell /cases/03583861 drwxrwxrwx+ 3 yank yank 4096 Aug 10 12:20 0090-odf-mustgather-before-08102023.tar.gz -rw-rw-rw-+ 1 yank yank 76617 Aug 10 12:20 0100-rook-ceph-operator-pod.log -rw-rw-rw-+ 1 yank yank 600190 Aug 10 12:20 0110-noobaa-core-0-pod.log -rw-rw-rw-+ 1 yank yank 13824 Aug 10 12:20 0120-before-collected-logs.txt -rw-rw-rw-+ 1 yank yank 1060086 Aug 10 12:20 0130-noobaa-operator-pod.log drwxrwxrwx+ 3 yank yank 4096 Aug 10 12:25 0140-odf-mustgather-after-08102023.tar.gz -rw-rw-rw-+ 1 yank yank 5304002 Aug 10 12:25 0150-noobaa-operator.log -rw-rw-rw-+ 1 yank yank 192344 Aug 10 12:25 0160-rook-ceph-operator-pod.log -rw-rw-rw-+ 1 yank yank 625848 Aug 10 12:25 0170-after-collected-logs.txt -rw-rw-rw-+ 1 yank yank 1190257 Aug 10 12:25 0180-noobaa-core-0-pod.log