Bug 2061713 - [KMS] The error message during creation of encrypted PVC mentions the parameter in UPPER_CASE
Summary: [KMS] The error message during creation of encrypted PVC mentions the paramet...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: csi-driver
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ODF 4.11.0
Assignee: Rakshith
QA Contact: Rachael
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-08 11:48 UTC by Rachael
Modified: 2023-08-09 16:37 UTC (History)
5 users (show)

Fixed In Version: 4.11.0-85
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-24 13:49:08 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ceph ceph-csi pull 3081 0 None open rbd: use `vaultAuthPath` variable name in error msg 2022-05-04 12:00:22 UTC
Github ceph ceph-csi pull 3138 0 None open rbd: use vaultAuthPath variable name in error msg 2022-05-26 06:59:45 UTC
Github red-hat-storage ceph-csi pull 94 0 None Merged Sync upstream devel to downstream devel 2022-05-05 10:30:56 UTC
Github red-hat-storage ceph-csi pull 95 0 None open Bug 2061713: rbd: use vaultAuthPath variable name in error msg 2022-05-26 07:41:27 UTC
Red Hat Product Errata RHSA-2022:6156 0 None None None 2022-08-24 13:49:35 UTC

Description Rachael 2022-03-08 11:48:19 UTC
Description of problem (please be detailed as possible and provide log
snippets):

The creation of encrypted RBD PVCs using kubernetes authentication method (using serviceaccount) fails with the following error:

$ oc describe pvc rbd-2
Name:          rbd-2
Namespace:     test-4
StorageClass:  test-pv-encryption-4
Status:        Pending
[...]

  Warning  ProvisioningFailed    75s (x10 over 5m31s)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-d98d9b847-vkhsn_031127f8-bf07-4c80-8a66-4e572b015807  failed to provision volume with StorageClass "test-pv-encryption-4": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed to set VAULT_AUTH_MOUNT_PATH in Vault config: path is empty

This error was seen even though the kubernetes authentication method was enabled at the default kubernetes/ path in Vault. 

$ oc get sc test-pv-encryption-4 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2022-03-08T11:07:13Z"
  name: test-pv-encryption-4
[...]
  encrypted: "true"
  encryptionKMSID: vault-sa-internal


$ oc get cm csi-kms-connection-details -n openshift-storage -o yaml
apiVersion: v1
data:
[...]
  vault-sa-internal: '{"encryptionKMSType":"vaulttenantsa","kmsServiceName":"vault-sa-internal","vaultAddress":"https://vault.default.svc.cluster.local:8200", "vaultBackendPath":"odf", "vaultTLSServerName":"","vaultNamespace":"","vaultCAFromSecret":"ocs-kms-ca-secret-4fjlyb","vaultCAFileName":"","vaultClientCertFileName":"", "vaultClientCertKeyFileName":"","vaultAuthMethod":"kubernetes","vaultAuthPath":"","vaultAuthNamespace":""}'


Commands run to configure the kubernetes authentication method in Vault:

    $ oc -n default exec -it vault-0 -- vault write -ca-cert /vault/userconfig/vault-server-tls/vault.crt auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"

    $ oc -n default exec -it vault-0 -- vault write -ca-cert /vault/userconfig/vault-server-tls/vault.crt auth/kubernetes/role/csi-kubernetes bound_service_account_names=ceph-csi-vault-sa bound_service_account_namespaces=test-4 



Version of all relevant components (if applicable):
---------------------------------------------------
OCP: 4.10.0-0.nightly-2022-03-08-002944
ODF: odf-operator.v4.10.0    full_version=4.10.0-179


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, creation of encrypted PVC fails


Is there any workaround available to the best of your knowledge?

Setting the vaultAuthPath in the csi-kms-connection-details configmap:

  vault-sa-internal: '{"encryptionKMSType":"vaulttenantsa","kmsServiceName":"vault-sa-internal","vaultAddress":"https://vault.default.svc.cluster.local:8200", "vaultBackendPath":"odf", "vaultTLSServerName":"","vaultNamespace":"","vaultCAFromSecret":"ocs-kms-ca-secret-4fjlyb","vaultCAFileName":"","vaultClientCertFileName":"", "vaultClientCertKeyFileName":"","vaultAuthMethod":"kubernetes","vaultAuthPath":"/v1/auth/kubernetes/login","vaultAuthNamespace":""}'



Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2


Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
N/A

If this is a regression, please provide more details to justify this:
Yes. For encrypted PVCs if the kubernetes authentication method was enabled at the default kubernetes/ path in Vault, then the vaultAuthPath didn't have to be explicitly specified in the KMS connection details.


Steps to Reproduce:
-------------------
1. Follow the steps to create an encrypted RBD PVC using vaulttenantsa encryptionKMSType as mentioned here: https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.9/html/managing_and_allocating_storage_resources/storage-classes_rhodf#prerequisites_for_using_literal_vaulttenantsa_literal

2. Check the status of the PVC


Actual results:
---------------

PVC is in pending state with the following error:

failed to provision volume with StorageClass "test-pv-encryption-4": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed to set VAULT_AUTH_MOUNT_PATH in Vault config: path is empty


Expected results:
-----------------
PVC should be in Bound state. Since the kubernetes authentication method is enabled in the default path, there should be no need to specify the authentication path.


Additional info:
----------------
The VAULT_AUTH_MOUNT_PATH was recently introduced in the UI for clusterwide encryption with this bug fix: https://bugzilla.redhat.com/show_bug.cgi?id=2048442

Comment 12 errata-xmlrpc 2022-08-24 13:49:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.11.0 security, enhancement, & bugfix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6156


Note You need to log in before you can comment on or make changes to this bug.