Created attachment 1903505 [details] Video showing the issue Description of problem (please be detailed as possible and provide log snippests): I updated the default backing store with the wrong secrets and the status of the backing store changed to Phase:Rejected. after this I created a pod and ran I/O on the BC with the default backing store and the I/O ran successfully. I am also able to create new BC even though the status is in Phase:Rejected State Version of all relevant components (if applicable): OCP 4.11 ODF 4.11 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes Is there any workaround available to the best of your knowledge? I do not know Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Login to OCP, go to Storage > Data Foundation > Backing store 2. Choose the default backing store "noobaa-default-backing-store" 3. Go to YAML and edit it with wrong secrets name. In my created a dummy secret and added its name here 4. After saving the YAML the status will change to "Rejected" 5. Now if I create a pod to run I/O, and attach a bucket claim to run the I/O. The I/O will run successfully. Actual results: The I/O is running successfully Expected results: The I/O should not run. Additional info: [sdurgbun@sdurgbun auth]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-07-29-173905 True False 2d1h Cluster version is 4.11.0-0.nightly-2022-07-29-173905 [sdurgbun@sdurgbun auth]$ [sdurgbun@sdurgbun auth]$ oc get pods -n openshift-storage NAME READY STATUS RESTARTS AGE csi-addons-controller-manager-7f8cb4f4bb-spwr8 2/2 Running 160 (50m ago) 2d1h csi-cephfsplugin-9hqns 3/3 Running 0 2d1h csi-cephfsplugin-cbbm2 3/3 Running 0 2d1h csi-cephfsplugin-mdck9 3/3 Running 0 2d1h csi-cephfsplugin-provisioner-7dc5f9468f-5l4cl 6/6 Running 604 (47m ago) 2d1h csi-cephfsplugin-provisioner-7dc5f9468f-7jcth 6/6 Running 607 (47m ago) 2d1h csi-rbdplugin-7sfkt 4/4 Running 0 2d1h csi-rbdplugin-98dzs 4/4 Running 0 2d1h csi-rbdplugin-kp284 4/4 Running 0 2d1h csi-rbdplugin-provisioner-64558c45f4-kjpm2 7/7 Running 604 (47m ago) 2d1h csi-rbdplugin-provisioner-64558c45f4-pk58c 7/7 Running 620 (47m ago) 2d1h nbio-1-759c8bc47f-v9rmz 1/1 Running 11 (6m21s ago) 116m nbio-1-776858bf88-gn5bw 0/1 CreateContainerConfigError 0 116m nbio-57ff79c568-d2bhj 1/1 Running 3 (10m ago) 125m noobaa-core-0 1/1 Running 0 45h noobaa-db-pg-0 1/1 Running 0 39h noobaa-endpoint-944fc767-lmtnw 1/1 Running 0 100m noobaa-operator-65fb6797bb-zp5x4 1/1 Running 0 100m ocs-metrics-exporter-9f5c496c7-45q7n 1/1 Running 0 2d1h ocs-operator-6d66f45447-98pnp 1/1 Running 203 (48m ago) 2d1h odf-console-7bbff96d7b-k7frh 1/1 Running 0 2d1h odf-operator-controller-manager-6c8d595449-z9xmh 2/2 Running 163 (48m ago) 2d1h rook-ceph-crashcollector-compute-0-7659ddb6ff-9mglw 1/1 Running 0 2d1h rook-ceph-crashcollector-compute-1-684b47fd48-nrr84 1/1 Running 0 100m rook-ceph-crashcollector-compute-2-54b7f57497-cwf98 1/1 Running 0 2d1h rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-698bcb4dpk4bk 2/2 Running 0 2d1h rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-599cddc98cqgn 2/2 Running 0 101m rook-ceph-mgr-a-7d755b549d-rmhgv 2/2 Running 0 2d1h rook-ceph-mon-a-7fc54b7675-75djf 2/2 Running 0 101m rook-ceph-mon-b-59d4b94d79-97q8q 2/2 Running 0 2d1h rook-ceph-mon-c-86ddf8859c-zr69k 2/2 Running 0 2d1h rook-ceph-operator-6478c668cc-c5n7l 1/1 Running 0 2d1h rook-ceph-osd-0-7c8bbf5545-xm669 2/2 Running 0 2d1h rook-ceph-osd-1-f68b8bdbf-5m6v9 2/2 Running 0 101m rook-ceph-osd-2-85b9877d54-kszq6 2/2 Running 0 2d1h rook-ceph-osd-prepare-ocs-deviceset-0-data-0qwxmx-v5chc 0/1 Completed 0 2d1h rook-ceph-osd-prepare-ocs-deviceset-1-data-0xb6wj-fdsf8 0/1 Completed 0 2d1h rook-ceph-osd-prepare-ocs-deviceset-2-data-0m8gr8-psvbs 0/1 Completed 0 2d1h rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-7c579d5rvlt4 2/2 Running 0 2d1h rook-ceph-tools-654c4cd949-5thd8 1/1 Running 0 2d1h s3cli-0 1/1 Running 0 19h [sdurgbun@sdurgbun auth]$ [sdurgbun@sdurgbun auth]$ [sdurgbun@sdurgbun auth]$ oc get csv --show-labels -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE LABELS mcg-operator.v4.11.0 NooBaa Operator 4.11.0 Succeeded operators.coreos.com/mcg-operator.openshift-storage= ocs-operator.v4.11.0 OpenShift Container Storage 4.11.0 Succeeded full_version=4.11.0-129,operatorframework.io/arch.amd64=supported,operatorframework.io/arch.ppc64le=supported,operatorframework.io/arch.s390x=supported,operators.coreos.com/ocs-operator.openshift-storage= odf-csi-addons-operator.v4.11.0 CSI Addons 4.11.0 Succeeded operators.coreos.com/odf-csi-addons-operator.openshift-storage= odf-operator.v4.11.0 OpenShift Data Foundation 4.11.0 Succeeded full_version=4.11.0-129,operatorframework.io/arch.amd64=supported,operatorframework.io/arch.ppc64le=supported,operatorframework.io/arch.s390x=supported,operators.coreos.com/odf-operator.openshift-storage= [sdurgbun@sdurgbun auth]$ [sdurgbun@sdurgbun auth]$ [sdurgbun@sdurgbun auth]$ oc get backingstore noobaa-default-backing-store -oyaml apiVersion: noobaa.io/v1alpha1 kind: BackingStore metadata: annotations: rgw: "" creationTimestamp: "2022-08-02T06:55:58Z" finalizers: - noobaa.io/finalizer generation: 14 labels: app: noobaa name: noobaa-default-backing-store namespace: openshift-storage ownerReferences: - apiVersion: noobaa.io/v1alpha1 blockOwnerDeletion: true controller: true kind: NooBaa name: noobaa uid: efbe36ae-02c9-4e99-bcc5-1178ae525df8 resourceVersion: "4398420" uid: cfa59222-c047-4c8a-9107-dc90a44a12ae spec: s3Compatible: endpoint: https://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc:443 secret: name: rook-ceph-object-user-ocs-storagecluster-cephobjectstore-noobaa-ceph-objectstore-user namespace: test signatureVersion: v4 targetBucket: nb.1659423358726.apps.sdurgbun-bz.qe.rh-ocs.com type: s3-compatible status: conditions: - lastHeartbeatTime: "2022-08-02T06:55:58Z" lastTransitionTime: "2022-08-04T08:08:56Z" message: BackingStore "noobaa-default-backing-store" invalid external connection "INVALID_CREDENTIALS" reason: INVALID_CREDENTIALS status: Unknown type: Available - lastHeartbeatTime: "2022-08-02T06:55:58Z" lastTransitionTime: "2022-08-04T08:08:56Z" message: BackingStore "noobaa-default-backing-store" invalid external connection "INVALID_CREDENTIALS" reason: INVALID_CREDENTIALS status: "False" type: Progressing - lastHeartbeatTime: "2022-08-02T06:55:58Z" lastTransitionTime: "2022-08-04T08:08:56Z" message: BackingStore "noobaa-default-backing-store" invalid external connection "INVALID_CREDENTIALS" reason: INVALID_CREDENTIALS status: "True" type: Degraded - lastHeartbeatTime: "2022-08-02T06:55:58Z" lastTransitionTime: "2022-08-04T08:08:56Z" message: BackingStore "noobaa-default-backing-store" invalid external connection "INVALID_CREDENTIALS" reason: INVALID_CREDENTIALS status: Unknown type: Upgradeable mode: modeCode: OPTIMAL timeStamp: 2022-08-02 10:45:03.799146713 +0000 UTC m=+286.942314118 phase: Rejected [sdurgbun@sdurgbun auth]$
Can this be fixed before dev freeze (15 nov) or should it be moved to 4.13?
Not marked as blocker, and we won't make it till frz. Moving out, can fix in a zstream once we fix in master
Closing since we cant do anything without logs, if this happens again, please collect ODF MG and reopen