Bug 2127027

Summary: [KMIP] PVC creation using encrypted RBD SC created during deployment fails
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Rachael <rgeorge>
Component: management-consoleAssignee: Sanjal Katiyar <skatiyar>
Status: CLOSED ERRATA QA Contact: Rachael <rgeorge>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.12CC: jefbrown, madam, mmuench, muagarwa, nthomas, ocs-bugs, odf-bz-bot, rar, skatiyar
Target Milestone: ---   
Target Release: ODF 4.12.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.12.0-61 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-31 00:19:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rachael 2022-09-15 06:43:19 UTC
Description of problem (please be detailed as possible and provide log
snippets):

In ODF 4.12, when storageclass encryption using Thales (KMIP) is selected during storagesystem creation, a new encrypted storageclass is created. 

The PVC creation using that storageclass fails with the following error:

$ oc describe pvc rbd-1
Name:          rbd-1
Namespace:     openshift-storage
StorageClass:  ocs-storagecluster-ceph-rbd-encrypted
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Used By:       <none>
Events:
  Type     Reason                Age                  From                                                                                                                Message
  ----     ------                ----                 ----                                                                                                                -------
  Normal   Provisioning          2m8s (x13 over 15m)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-7db859c675-7frmz_c1d0e2c3-07b8-4313-a01a-6b71b8fa4c25  External provisioner is provisioning volume for claim "openshift-storage/rbd-1"
  Warning  ProvisioningFailed    2m8s (x13 over 15m)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-7db859c675-7frmz_c1d0e2c3-07b8-4313-a01a-6b71b8fa4c25  failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd-encrypted": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed to get secrets: unsupported option for KMS provider "kmip": UniqueIdentifier


$ oc get sc ocs-storagecluster-ceph-rbd-encrypted -o yaml | grep encrypt
  name: ocs-storagecluster-ceph-rbd-encrypted
  encrypted: "true"
  encryptionKMSID: ciphertrust


$ oc get cm csi-kms-connection-details -o yaml
apiVersion: v1
data:
  ciphertrust: '{"KMS_PROVIDER":"kmip","KMS_SERVICE_NAME":"ciphertrust","KMIP_ENDPOINT":"x.x.x.x:5697","KMIP_SECRET_NAME":"thales-kmip-kms-blid9f","TLS_SERVER_NAME":"kmip_all_5697.ciphertrustmanager.local"}'


$ oc get secret thales-kmip-kms-blid9f -o yaml
apiVersion: v1
data:
  CA_CERT: [...]
  CLIENT_CERT: [...]
  CLIENT_KEY: [...]
  UNIQUE_IDENTIFIER: NWQ0MmU0NTRiMmUwNDBiNWEzOGVmY...
  UniqueIdentifier: NzlmOTkyMTlkNzkxNGM3M2F...
kind: Secret


$ oc get storagecluster -o yaml
  spec:
    arbiter: {}
    encryption:
      clusterWide: true
      enable: true
      kms:
        enable: true
      storageClass: true



Version of all relevant components (if applicable):
---------------------------------------------------
OCP: 4.12.0-0.nightly-2022-09-08-114806
ODF: odf-operator.v4.12.0    full_version=4.12.0-53



Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, the encryption enabled SC created during deployment is not usable

Is there any workaround available to the best of your knowledge?
Create a new custom encryption enabled RBD SC

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
No

Steps to Reproduce:
1. Install ODF 4.12 operator
2. During storagesystem creation enable both clusterwide and storageclass encryption
3. Create a PVC using the ocs-storagecluster-ceph-rbd-encrypted SC


Actual results:
---------------

PVC is stuck in pending state with the following error:

  Warning  ProvisioningFailed    2m8s (x13 over 15m)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-7db859c675-7frmz_c1d0e2c3-07b8-4313-a01a-6b71b8fa4c25  failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd-encrypted": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed to get secrets: unsupported option for KMS provider "kmip": UniqueIdentifier


Expected results:
-----------------

PVC creation should be successful

Comment 22 errata-xmlrpc 2023-01-31 00:19:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.12.0 enhancement and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:0551