Bug 2115241

Summary: Still able to run IO and use bucket claims even after giving wrong secrets to the backing store
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Shivam Durgbuns <sdurgbun>
Component: Multi-Cloud Object GatewayAssignee: Sheetal Pamecha <spamecha>
Status: CLOSED INSUFFICIENT_DATA QA Contact: krishnaram Karthick <kramdoss>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.11CC: kramdoss, muagarwa, nbecker, ocs-bugs, odf-bz-bot, spamecha
Target Milestone: ---   
Target Release: ODF 4.14.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-19 08:09:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Video showing the issue none

Description Shivam Durgbuns 2022-08-04 08:22:05 UTC 
Created attachment 1903505 [details]
Video showing the issue

Description of problem (please be detailed as possible and provide log
snippests):
I updated the default backing store with the wrong secrets and the status of the  backing store changed to Phase:Rejected. after this I created a pod and ran I/O on the BC with the default backing store and the I/O ran successfully. I am also able to create new BC even though the status is in Phase:Rejected State

Version of all relevant components (if applicable):
OCP 4.11 ODF 4.11

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes


Is there any workaround available to the best of your knowledge?
I do not know

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Login to OCP, go to Storage > Data Foundation > Backing store
2. Choose the default backing store "noobaa-default-backing-store"
3. Go to YAML and edit it with wrong secrets name. In my created a dummy secret and added its name here
4. After saving the YAML the status will change to "Rejected"
5. Now if I create a pod to run I/O, and attach a bucket claim to run the I/O. The I/O will run successfully.


Actual results:
The I/O is running successfully


Expected results:
The I/O should not run.

Additional info:
[sdurgbun@sdurgbun auth]$  oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-07-29-173905   True        False         2d1h    Cluster version is 4.11.0-0.nightly-2022-07-29-173905
[sdurgbun@sdurgbun auth]$ 
[sdurgbun@sdurgbun auth]$ oc get pods -n openshift-storage
NAME                                                              READY   STATUS                       RESTARTS         AGE
csi-addons-controller-manager-7f8cb4f4bb-spwr8                    2/2     Running                      160 (50m ago)    2d1h
csi-cephfsplugin-9hqns                                            3/3     Running                      0                2d1h
csi-cephfsplugin-cbbm2                                            3/3     Running                      0                2d1h
csi-cephfsplugin-mdck9                                            3/3     Running                      0                2d1h
csi-cephfsplugin-provisioner-7dc5f9468f-5l4cl                     6/6     Running                      604 (47m ago)    2d1h
csi-cephfsplugin-provisioner-7dc5f9468f-7jcth                     6/6     Running                      607 (47m ago)    2d1h
csi-rbdplugin-7sfkt                                               4/4     Running                      0                2d1h
csi-rbdplugin-98dzs                                               4/4     Running                      0                2d1h
csi-rbdplugin-kp284                                               4/4     Running                      0                2d1h
csi-rbdplugin-provisioner-64558c45f4-kjpm2                        7/7     Running                      604 (47m ago)    2d1h
csi-rbdplugin-provisioner-64558c45f4-pk58c                        7/7     Running                      620 (47m ago)    2d1h
nbio-1-759c8bc47f-v9rmz                                           1/1     Running                      11 (6m21s ago)   116m
nbio-1-776858bf88-gn5bw                                           0/1     CreateContainerConfigError   0                116m
nbio-57ff79c568-d2bhj                                             1/1     Running                      3 (10m ago)      125m
noobaa-core-0                                                     1/1     Running                      0                45h
noobaa-db-pg-0                                                    1/1     Running                      0                39h
noobaa-endpoint-944fc767-lmtnw                                    1/1     Running                      0                100m
noobaa-operator-65fb6797bb-zp5x4                                  1/1     Running                      0                100m
ocs-metrics-exporter-9f5c496c7-45q7n                              1/1     Running                      0                2d1h
ocs-operator-6d66f45447-98pnp                                     1/1     Running                      203 (48m ago)    2d1h
odf-console-7bbff96d7b-k7frh                                      1/1     Running                      0                2d1h
odf-operator-controller-manager-6c8d595449-z9xmh                  2/2     Running                      163 (48m ago)    2d1h
rook-ceph-crashcollector-compute-0-7659ddb6ff-9mglw               1/1     Running                      0                2d1h
rook-ceph-crashcollector-compute-1-684b47fd48-nrr84               1/1     Running                      0                100m
rook-ceph-crashcollector-compute-2-54b7f57497-cwf98               1/1     Running                      0                2d1h
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-698bcb4dpk4bk   2/2     Running                      0                2d1h
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-599cddc98cqgn   2/2     Running                      0                101m
rook-ceph-mgr-a-7d755b549d-rmhgv                                  2/2     Running                      0                2d1h
rook-ceph-mon-a-7fc54b7675-75djf                                  2/2     Running                      0                101m
rook-ceph-mon-b-59d4b94d79-97q8q                                  2/2     Running                      0                2d1h
rook-ceph-mon-c-86ddf8859c-zr69k                                  2/2     Running                      0                2d1h
rook-ceph-operator-6478c668cc-c5n7l                               1/1     Running                      0                2d1h
rook-ceph-osd-0-7c8bbf5545-xm669                                  2/2     Running                      0                2d1h
rook-ceph-osd-1-f68b8bdbf-5m6v9                                   2/2     Running                      0                101m
rook-ceph-osd-2-85b9877d54-kszq6                                  2/2     Running                      0                2d1h
rook-ceph-osd-prepare-ocs-deviceset-0-data-0qwxmx-v5chc           0/1     Completed                    0                2d1h
rook-ceph-osd-prepare-ocs-deviceset-1-data-0xb6wj-fdsf8           0/1     Completed                    0                2d1h
rook-ceph-osd-prepare-ocs-deviceset-2-data-0m8gr8-psvbs           0/1     Completed                    0                2d1h
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-7c579d5rvlt4   2/2     Running                      0                2d1h
rook-ceph-tools-654c4cd949-5thd8                                  1/1     Running                      0                2d1h
s3cli-0                                                           1/1     Running                      0                19h
[sdurgbun@sdurgbun auth]$ 
[sdurgbun@sdurgbun auth]$ 
[sdurgbun@sdurgbun auth]$ oc get csv --show-labels -n openshift-storage
NAME                              DISPLAY                       VERSION   REPLACES   PHASE       LABELS
mcg-operator.v4.11.0              NooBaa Operator               4.11.0               Succeeded   operators.coreos.com/mcg-operator.openshift-storage=
ocs-operator.v4.11.0              OpenShift Container Storage   4.11.0               Succeeded   full_version=4.11.0-129,operatorframework.io/arch.amd64=supported,operatorframework.io/arch.ppc64le=supported,operatorframework.io/arch.s390x=supported,operators.coreos.com/ocs-operator.openshift-storage=
odf-csi-addons-operator.v4.11.0   CSI Addons                    4.11.0               Succeeded   operators.coreos.com/odf-csi-addons-operator.openshift-storage=
odf-operator.v4.11.0              OpenShift Data Foundation     4.11.0               Succeeded   full_version=4.11.0-129,operatorframework.io/arch.amd64=supported,operatorframework.io/arch.ppc64le=supported,operatorframework.io/arch.s390x=supported,operators.coreos.com/odf-operator.openshift-storage=
[sdurgbun@sdurgbun auth]$ 
[sdurgbun@sdurgbun auth]$ 
[sdurgbun@sdurgbun auth]$ oc get backingstore noobaa-default-backing-store -oyaml
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  annotations:
    rgw: ""
  creationTimestamp: "2022-08-02T06:55:58Z"
  finalizers:
  - noobaa.io/finalizer
  generation: 14
  labels:
    app: noobaa
  name: noobaa-default-backing-store
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: noobaa.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: NooBaa
    name: noobaa
    uid: efbe36ae-02c9-4e99-bcc5-1178ae525df8
  resourceVersion: "4398420"
  uid: cfa59222-c047-4c8a-9107-dc90a44a12ae
spec:
  s3Compatible:
    endpoint: https://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc:443
    secret:
      name: rook-ceph-object-user-ocs-storagecluster-cephobjectstore-noobaa-ceph-objectstore-user
      namespace: test
    signatureVersion: v4
    targetBucket: nb.1659423358726.apps.sdurgbun-bz.qe.rh-ocs.com
  type: s3-compatible
status:
  conditions:
  - lastHeartbeatTime: "2022-08-02T06:55:58Z"
    lastTransitionTime: "2022-08-04T08:08:56Z"
    message: BackingStore "noobaa-default-backing-store" invalid external connection
      "INVALID_CREDENTIALS"
    reason: INVALID_CREDENTIALS
    status: Unknown
    type: Available
  - lastHeartbeatTime: "2022-08-02T06:55:58Z"
    lastTransitionTime: "2022-08-04T08:08:56Z"
    message: BackingStore "noobaa-default-backing-store" invalid external connection
      "INVALID_CREDENTIALS"
    reason: INVALID_CREDENTIALS
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2022-08-02T06:55:58Z"
    lastTransitionTime: "2022-08-04T08:08:56Z"
    message: BackingStore "noobaa-default-backing-store" invalid external connection
      "INVALID_CREDENTIALS"
    reason: INVALID_CREDENTIALS
    status: "True"
    type: Degraded
  - lastHeartbeatTime: "2022-08-02T06:55:58Z"
    lastTransitionTime: "2022-08-04T08:08:56Z"
    message: BackingStore "noobaa-default-backing-store" invalid external connection
      "INVALID_CREDENTIALS"
    reason: INVALID_CREDENTIALS
    status: Unknown
    type: Upgradeable
  mode:
    modeCode: OPTIMAL
    timeStamp: 2022-08-02 10:45:03.799146713 +0000 UTC m=+286.942314118
  phase: Rejected
[sdurgbun@sdurgbun auth]$
Comment 9 Mudit Agarwal 2022-10-31 03:07:04 UTC 
Can this be fixed before dev freeze (15 nov) or should it be moved to 4.13?
Comment 10 Nimrod Becker 2022-11-03 07:40:48 UTC 
Not marked as blocker, and we won't make it till frz.
Moving out, can fix in a zstream once we fix in master
Comment 27 Nimrod Becker 2023-07-19 08:09:04 UTC 
Closing since we cant do anything without logs, if this happens again, please collect ODF MG and reopen