Description of problem:

- I was Trying to test the Noobaa Encryption with HPCS,  I configured the backingStore as IBM-COS and created bucket class, storage class, objectbucketClaim, POD and did some read/write of Data.
- I could see the Data getting uploaded to ibm-cos and i could see DMS files for each object i pushed. Then i BLOCKED the HPCS ROOT Key , that used for encryption HPCS console. and Try to Read/Write Data again from inside the POD,
- I was expecting the Upload/download will fail..but that was not case. 
Let me know, if the above USECASE is VALID or there are any ISSUE in the test/bug??? 

Version-Release number of selected component (if applicable): 4.10.0-158 

How reproducible:

Steps to Reproduce:

1. I installed ODF build 4.10.0-158 on my IBM cluster 
2. I have selected the Cluster encryption to use HPCS during the ODF install
3. I see the noobaa PODs are running fine.
$ oc get pods | grep noobaa
noobaa-core-0                                                     1/1     Running       0                26d
noobaa-db-pg-0                                                    1/1     Running       0                26d
noobaa-default-backing-store-noobaa-pod-7c0d7cfc                  1/1     Running       760 (137m ago)   9d
noobaa-endpoint-5dcd5f8dc9-ldltc                                  1/1     Running       0                26d
noobaa-operator-859fb4fb57-94rbb                                  1/1     Running       1 (4d7h ago)     6d3h

4. I also see the noobaa CR has the encryption parameters inside
$ oc describe noobaa 

Security:
    Kms:
      Connection Details:
        IBM_KP_BASE_URL:             https://api.au-syd.hs-crypto.cloud.ibm.com:9884
        IBM_KP_SECRET_NAME:          ibm-kp-kms-cdiwr2
        IBM_KP_SERVICE_INSTANCE_ID:  b6666551-1688-4937-b165-d5424087fb50
        IBM_KP_TOKEN_URL:            
        KMS_PROVIDER:                ibmkeyprotect
        KMS_SERVICE_NAME:            myhpcconnect1
      Token Secret Name:             ibm-kp-kms-cdiwr2

5. Then i tried to create a backingstore pointing to IBM-COS as per the DOC

 https://github.com/noobaa/noobaa-operator/blob/master/doc/backing-store-crd.md#ibm-cos-type

$ oc get backingstore
NAME                           TYPE      PHASE   AGE
bs                             ibm-cos   Ready   5d23h


6. Then created a bucket class pointing to the imm-cos backing device
$ oc get bucketclass
NAME                          PLACEMENT                                                        NAMESPACEPOLICY   QUOTA   PHASE   AGE
ibmcos-bucket-class           {"tiers":[{"backingStores":["bs"]}]}                                                       Ready   5d23h

7. Then Created a SC which uses ibmcos-bucket-class
$ oc get sc
NAME                                    PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ibmcos-noobaa.noobaa.io                 openshift-storage.noobaa.io/obc         Delete          Immediate              false                  5d23h

8. Then created an OBC with the above storage class ibmcos-noobaa.noobaa.io
$ oc get obc
NAME             STORAGE-CLASS                 PHASE   AGE
ibmcos-obc1      ibmcos-noobaa.noobaa.io       Bound   5d23h

9. Then i see the OB  Object and the Secret and Config Resources

$ oc get ob
NAME                                   STORAGE-CLASS                 CLAIM-NAMESPACE   CLAIM-NAME   RECLAIM-POLICY   PHASE   AGE
obc-openshift-storage-ibmcos-obc1      ibmcos-noobaa.noobaa.io                                      Delete           Bound   5d23h 


$ oc get secret 

ibmcos-obc1                                                                                       Opaque                                2      5d23h

$ oc get cm
ibmcos-obc1                                                   5      5d23h


10. Then created a POD which uses the above OBC secret and OBC config
$ oc get pod 
pod-devtest-ibm-cos                                               1/1     Running       0                2d2h

$ oc describe pod pod-devtest-ibm-cos
..
    Environment Variables from:
      ibmcos-obc1  Secret     Optional: false
      ibmcos-obc1  ConfigMap  Optional: false
    Environment:   <none>

11. then exec inside POD and wrote some data

$ oc  exec -it pod-devtest-ibm-cos -- sh
/ # pip install awscli-plugin-endpoint;
/ # aws configure set plugins.endpoint awscli_plugin_endpoint;
/ # aws configure set s3.endpoint_url https://s3.openshift-storage.svc
/ # aws configure set ca_bundle /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt;
/ # aws s3 ls s3://
2022-03-16 12:12:11 my-bucket-eu-gb1
/ # aws s3 cp ./test123.txt s3://my-bucket-eu-gb1/
upload: test123.txt to s3://my-bucket-eu-gb1/test123.txt
/ # aws s3 ls s3://my-bucket-eu-gb1/
2022-03-10 15:08:40  114  test123.txt


12. Next , i DISABLED the HPCS key IBM_KP_CUSTOMER_ROOT_KEY from IBM console, that's being used by noobaa (in Step4)

        IBM_KP_SECRET_NAME:          ibm-kp-kms-cdiwr2
        IBM_KP_SERVICE_INSTANCE_ID:  b6666551-1688-4937-b165-d5424087fb50
        IBM_KP_TOKEN_URL:            
        KMS_PROVIDER:                ibmkeyprotect
        KMS_SERVICE_NAME:            myhpcconnect1

13. Then I try to read/write some Data gain via POD (as in Step 11). 
# aws s3 cp ./dddd.txt s3://my-bucket-eu-gb1/
upload: ./dddd.txt to s3://my-bucket-eu-gb1/dddd.txt

# aws s3 ls s3://my-bucket-eu-gb1/
dddd.txt


But I was expecting the upload/Read to FAIL because of HPCS root KEY disable, but that worked.


Actual results: When HPCS key is DISABLED, Still the POD can do read/write to the s3 nooba Bucket


Expected results: When HPCS key is DISABLED, further read/write to the Bucket should Fail. As cluster is using the HPCS root key for encryption.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info: