Bug 2277950
| Summary: | noobaa-default-backing-store in Rejected state in MCG Only deployment | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Vijay Avuthu <vavuthu> |
| Component: | Multi-Cloud Object Gateway | Assignee: | Nimrod Becker <nbecker> |
| Status: | CLOSED DUPLICATE | QA Contact: | krishnaram Karthick <kramdoss> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.16 | CC: | dzaken, odf-bz-bot |
| Target Milestone: | --- | Keywords: | Automation, Regression |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-05-07 08:52:06 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 2278120 *** |
Description of problem (please be detailed as possible and provide log snippests): noobaa-default-backing-store in Rejected state in MCG Only deployment. Version of all relevant components (if applicable): ocs-operator.v4.16.0-90.stable Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? not able to install MCG only deployment Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes, 2/2 Can this issue reproduce from the UI? not tried If this is a regression, please provide more details to justify this: Yes Steps to Reproduce: 1. Install MCG only deployment using ocs-ci 2. check backingstore status 3. Actual results: $ oc get backingstore NAME TYPE PHASE AGE noobaa-default-backing-store pv-pool Rejected 31m Expected results: Additional info: $ oc describe backingstore noobaa-default-backing-store Name: noobaa-default-backing-store Namespace: openshift-storage Labels: app=noobaa Annotations: <none> API Version: noobaa.io/v1alpha1 Kind: BackingStore . . . Status: Conditions: Last Heartbeat Time: 2024-04-30T09:14:03Z Last Transition Time: 2024-04-30T09:14:03Z Message: Current failing: 1 from requested: 1 Reason: Failed connecting all pods in backingstore for more than 10 minutes Status: Unknown Type: Available Last Heartbeat Time: 2024-04-30T09:14:03Z Last Transition Time: 2024-04-30T09:14:03Z Message: Current failing: 1 from requested: 1 Reason: Failed connecting all pods in backingstore for more than 10 minutes Status: False Type: Progressing Last Heartbeat Time: 2024-04-30T09:14:03Z Last Transition Time: 2024-04-30T09:14:03Z Message: Current failing: 1 from requested: 1 Reason: Failed connecting all pods in backingstore for more than 10 minutes Status: True Type: Degraded Last Heartbeat Time: 2024-04-30T09:14:03Z Last Transition Time: 2024-04-30T09:14:03Z Message: Current failing: 1 from requested: 1 Reason: Failed connecting all pods in backingstore for more than 10 minutes Status: Unknown Type: Upgradeable > noobaa-default-backing-store-noobaa-pod is in CLBO $ oc get pods NAME READY STATUS RESTARTS AGE csi-addons-controller-manager-67499dd99d-5dcdl 2/2 Running 0 44m noobaa-core-0 2/2 Running 0 44m noobaa-db-pg-0 1/1 Running 0 44m noobaa-default-backing-store-noobaa-pod-8b0ee01e 0/1 CrashLoopBackOff 11 (3m17s ago) 42m noobaa-endpoint-7dcc44f954-4gl2x 1/1 Running 0 43m noobaa-operator-555c8784c8-5bng4 1/1 Running 0 46m ocs-client-operator-console-75fc966869-5zzw2 1/1 Running 0 45m ocs-client-operator-controller-manager-7944f448b5-5zzrp 2/2 Running 0 45m ocs-metrics-exporter-6558b7585b-dd6vf 1/1 Running 0 44m ocs-operator-776bbff6d4-r642w 1/1 Running 0 46m odf-console-6d446c5b48-ml46z 1/1 Running 0 46m odf-operator-controller-manager-699bdc69ff-6nhmn 2/2 Running 0 46m rook-ceph-operator-7f598695f5-pfjsn 1/1 Running 0 46m ux-backend-server-5f586bdd95-r6fww 2/2 Running 0 46m > $ oc describe pod noobaa-default-backing-store-noobaa-pod-8b0ee01e Name: noobaa-default-backing-store-noobaa-pod-8b0ee01e Namespace: openshift-storage Priority: 0 Service Account: default Node: j-024vi1rmcg33-t1-kbk95-worker-0-c6mwt/10.1.113.160 Start Time: Tue, 30 Apr 2024 14:03:05 +0530 Labels: app=noobaa pool=noobaa-default-backing-store . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 42m default-scheduler Successfully assigned openshift-storage/noobaa-default-backing-store-noobaa-pod-8b0ee01e to j-024vi1rmcg33-t1-kbk95-worker-0-c6mwt Normal SuccessfulAttachVolume 42m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-e9ad53fe-36ad-4192-9ac6-79b139409bd4" Normal AddedInterface 42m multus Add eth0 [10.131.0.23/23] from ovn-kubernetes Normal Pulled 38m (x5 over 42m) kubelet Container image "registry.redhat.io/odf4/mcg-core-rhel9@sha256:53f62e85f733d38b511b4b2f0ebb0da40e606c55f10c592781f751c960bf9b3c" already present on machine Normal Created 38m (x5 over 42m) kubelet Created container noobaa-agent Normal Started 38m (x5 over 42m) kubelet Started container noobaa-agent Warning BackOff 2m22s (x153 over 41m) kubelet Back-off restarting failed container noobaa-agent in pod noobaa-default-backing-store-noobaa-pod-8b0ee01e_openshift-storage(15b08b26-c3ae-426b-b3be-4c24d6df9bef) > noobaa operator log time="2024-04-30T09:05:59Z" level=info msg="SetPhase: Rejected" backingstore=openshift-storage/noobaa-default-backing-store time="2024-04-30T09:05:59Z" level=error msg="❌ Persistent Error: Current failing: 1 from requested: 1" backingstore=openshift-storage/noobaa-default-backing-store time="2024-04-30T09:05:59Z" level=info msg="Update event detected for noobaa-default-backing-store (openshift-storage), queuing Reconcile" time="2024-04-30T09:05:59Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore openshift-storage/noobaa-default-backing-store to bucketclasses" time="2024-04-30T09:05:59Z" level=info msg="UpdateStatus: Done" backingstore=openshift-storage/noobaa-default-backing-store time="2024-04-30T09:05:59Z" level=info msg="will reconcile these bucketclasses: [openshift-storage/noobaa-default-bucket-class]" time="2024-04-30T09:05:59Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore openshift-storage/noobaa-default-backing-store to bucketclasses" time="2024-04-30T09:05:59Z" level=info msg="Start BucketClass Reconcile..." bucketclass=openshift-storage/noobaa-default-bucket-class time="2024-04-30T09:05:59Z" level=info msg="will reconcile these bucketclasses: [openshift-storage/noobaa-default-bucket-class]" time="2024-04-30T09:05:59Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n" time="2024-04-30T09:05:59Z" level=info msg="✅ Exists: BucketClass \"noobaa-default-bucket-class\"\n" time="2024-04-30T09:05:59Z" level=info msg="SetPhase: Verifying" bucketclass=openshift-storage/noobaa-default-bucket-class time="2024-04-30T09:05:59Z" level=info msg="validating placement policy &{Tiers:[{Placement: BackingStores:[noobaa-default-backing-store]}]}" time="2024-04-30T09:05:59Z" level=info msg="✅ Exists: BackingStore \"noobaa-default-backing-store\"\n" time="2024-04-30T09:05:59Z" level=info msg="SetPhase: Rejected" bucketclass=openshift-storage/noobaa-default-bucket-class time="2024-04-30T09:05:59Z" level=error msg="❌ Persistent Error: NooBaa BackingStore \"noobaa-default-backing-store\" is in rejected phase" bucketclass=openshift-storage/noobaa-default-bucket-class time="2024-04-30T09:05:59Z" level=info msg="UpdateStatus: Done" bucketclass=openshift-storage/noobaa-default-bucket-class must gather: https://url.corp.redhat.com/02e543e job: https://url.corp.redhat.com/2789162