Bug 2097727

Summary: PVCs with AWS KMS are stuck due to missing secret rook-csi-rbd-provisioner on consumer
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Filip Balák <fbalak>
Component: odf-managed-serviceAssignee: Ohad <omitrani>
Status: CLOSED NOTABUG QA Contact: Neha Berry <nberry>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.11CC: aeyal, mrajanna, ocs-bugs, odf-bz-bot, sabose
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-16 12:33:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Filip Balák 2022-06-16 12:15:54 UTC 
Description of problem:
When KMS is configured on consumer cluster and PVC is created, the PVC is stuck in pending with event message:

```
Generated from openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-bd98cf759-n7spw_94092a20-ec6d-44ad-8cc3-114aa0a898dc

failed to provision volume with StorageClass "openshift-storage-block": error getting secret rook-csi-rbd-provisioner in namespace openshift-storage: secrets "rook-csi-rbd-provisioner" not found
```

This secret is only available on provider cluster and is missing on consumer:
$ oc get secret -n openshift-storage|grep rbd
rook-csi-rbd-plugin-sa-dockercfg-lqhgt               kubernetes.io/dockercfg               1      26h
rook-csi-rbd-plugin-sa-token-5lkkt                   kubernetes.io/service-account-token   4      26h
rook-csi-rbd-plugin-sa-token-rp7hh                   kubernetes.io/service-account-token   4      26h
rook-csi-rbd-provisioner-sa-dockercfg-q7jwv          kubernetes.io/dockercfg               1      26h
rook-csi-rbd-provisioner-sa-token-l9klr              kubernetes.io/service-account-token   4      26h
rook-csi-rbd-provisioner-sa-token-xmpzw              kubernetes.io/service-account-token   4      26h

Version-Release number of selected component (if applicable):
$ oc get csv -n openshift-storage
NAME                                      DISPLAY                       VERSION           REPLACES                                  PHASE
mcg-operator.v4.11.0                      NooBaa Operator               4.11.0            mcg-operator.v4.10.3                      Succeeded
ocs-operator.v4.11.0                      OpenShift Container Storage   4.11.0            ocs-operator.v4.10.3                      Succeeded
ocs-osd-deployer.v2.0.2                   OCS OSD Deployer              2.0.2             ocs-osd-deployer.v2.0.1                   Succeeded
odf-csi-addons-operator.v4.11.0           CSI Addons                    4.11.0            odf-csi-addons-operator.v4.10.3           Succeeded
odf-operator.v4.11.0                      OpenShift Data Foundation     4.11.0            odf-operator.v4.10.2                      Succeeded
ose-prometheus-operator.4.10.0            Prometheus Operator           4.10.0            ose-prometheus-operator.4.8.0             Succeeded
route-monitor-operator.v0.1.420-b65f47e   Route Monitor Operator        0.1.420-b65f47e   route-monitor-operator.v0.1.418-6459408   Succeeded


How reproducible:
1/1

Steps to Reproduce:
1. Configure KMS on consumer cluster according to https://hackmd.io/66K6Opp8RKGUlRsM62A_rg
2. Create namespace, kms secret in that namespace and a pvc that uses encrypted storageclass.
3. Check events of the pvc.

Actual results:
PVC is pending due to:
```
failed to provision volume with StorageClass "openshift-storage-block": error getting secret rook-csi-rbd-provisioner in namespace openshift-storage: secrets "rook-csi-rbd-provisioner" not found
```

Expected results:
PVC should get bound.

Additional info:
Steps to set kms: https://hackmd.io/66K6Opp8RKGUlRsM62A_rg
Steps to update addon OCS to 4.11 for KMS testing: https://docs.google.com/document/d/1gm-msSAEWsk-bM8Dvdd_IDHUJuf8RjEKoOS1k7q26cY/edit