Bug 2005801
| Summary: | [KMS] Tenant config does not override backendpath if the key is specified in UPPER_CASE | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Rachael <rgeorge> |
| Component: | management-console | Assignee: | gowtham <gshanmug> |
| Status: | CLOSED ERRATA | QA Contact: | Rachael <rgeorge> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.9 | CC: | afrahman, gshanmug, jefbrown, kbg, madam, muagarwa, nberry, ndevos, nthomas, ocs-bugs, odf-bz-bot, shan, shilpsha, skatiyar |
| Target Milestone: | --- | ||
| Target Release: | ODF 4.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
.OpenShift Data Foundation user interface available in camel case
Previously, OpenShift Data Foundation user interface used upper case to store the vault Key Management System (KMS) configs in the csi-kms-connection-details config map. However, Ceph Container Storage Interface (CSI) supports the upper case for a user interface at limited places. Ceph CSI recommends using camel cases in most places. As a result, the csi-kms-connection-details config map is mixed with both upper and lower cases which caused confusion.
With this update, the user interface is moving to the camel case while supporting the upper case for backward compatibility.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-04-13 18:49:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2011326, 2056571 | ||
It is intentional that the tenants ConfigMap `ceph-csi-kms-config` only can contain the camelCase keys. The UPPER_CASE keys were an addition only for the UI as mapping the user input to the pre-existing ceph-csi configuration options was deemed to complex and confusing. The camelCase configuration is the default, and offers more advanced options than the UI is expected to handle. I think this is not so much a bug in Ceph-CSI, but rather a documentation topic. Would that work for you, Rachael? Niels, should this be moved to UI? Adding it as a known issue and moving to the UI team. Till we decide for a fix, we can add it in release notes. Niels, please fill the doc text. This is indeed a limitation imposed by the UI. There is no assistance from the UI to configure a KMS with tenant specific options. The UI required only UPPER_CASE (environment like key/value) options to be supported, whereas Ceph-CSI uses camelCase formatting. The added functionality in Ceph-CSI to support UPPER_CASE options is very limited, the camelCase configuration format supports more advanced features. The tenant configuration for KMS providers is expected to be in camelCase (UI is not involved at all). Ideally the UI can support the camelCase configuration options, which will allow a better configuration for global setting that are related to tenants as well (these are not key/value environment like parameters). From a Rook perspective, we treat the VAULT flags as environment variables, which are all described here: https://www.vaultproject.io/docs/commands#environment-variables So I believe ceph-csi should comply to that format. @ndevos (In reply to Sébastien Han from comment #17) > From a Rook perspective, we treat the VAULT flags as environment variables, > which are all described here: > https://www.vaultproject.io/docs/commands#environment-variables > So I believe ceph-csi should comply to that format. Ceph-CSI does not use this as environment variables, but reads the options as a JSON configuration file. Some variables can be nested, or overridden for selected tenants/namespaces. The format for the variables was defined before the downstream product even considered supporting PV-encryption. The UI required formatting in UPPER_CASE, which was introduced for a subset of the available options. There are options that do not have matching environment variables, but are consumed by the API (e.g. VAULT_DESTROY_KEYS in libopenstorage/secrets, or VAULT_AUTH_NAMESPACE in Ceph-CSI). Technically Ceph-CSI should be able to add a mapping from (to be added) UPPER_CASE options to existing camelCase ones. This will require some work and quite some testing, the current implementation is not prepared for alias-names of options. As Ceph-CSI does offer more options than Vault (or Rook) does, the UPPER_CASE options will not be documented by the KMS provider (https://www.vaultproject.io/docs/commands#environment-variables). At the moment, I am not sure what is more confusing for users: a. a selection of 'environment' variables that are partially listed on the KMS product docs b. a configuration file (ConfigMap) containing Ceph-CSI specific options, making a clear distinction between what is supported and tested An other idea: As the Ceph-CSI configuration is in JSON, it is possible to annotate the sections with a note or warning, like vault-token-v1: '{ "_WARNING": "this section is generated by ODF, use UPPER_CASE options only", "KMS_PROVIDER": "vaulttokens", "KMS_SERVICE_NAME": "vault-token-v1", "VAULT_ADDR": "https://vault.ocs.com:8200", "VAULT_BACKEND_PATH": "vb-sys-v1", "VAULT_CACERT": "ocs-kms-ca-secret-izzsps", "VAULT_TLS_SERVER_NAME": "", "VAULT_CLIENT_CERT": "ocs-kms-client-cert-3jnpp", "VAULT_CLIENT_KEY": "ocs-kms-client-key-r1f8op", "VAULT_NAMESPACE": "", "VAULT_TOKEN_NAME": "", "VAULT_CACERT_FILE": "fullchain.pem", "VAULT_CLIENT_CERT_FILE": "cert.pem", "VAULT_CLIENT_KEY_FILE": "privkey.pem" }' Users will then be warned to not use camelCase options (which will be ignored) in that section. Please test with the latest build. Doc text? User need to with UI only, whether It is camel case or upper case will taken care by UI. Gowtham, you need to fill the doc text in the **Doc Text:** field of the bug. There is a format which needs to be followed while filling the doc text. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.10.0 enhancement, security & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:1372 |
Description of problem (please be detailed as possible and provide log snippets): In ODF 4.9, with RBD PV encryption, the vault backend path can be overridden in the tenant namespace by creating a new configmap and specifying the backend path to be used for storing the Key encryption keys for the encrypted PVs in the given tenant namespace. When the key in the configmap is specified using UPPER_CASE, ceph-csi still uses the backend path specified in the csi-kms-connection-details configmap in the openshift-storage namespace. Backend Path specified while creating the storageclass: ------------------------------------------------------- $ oc get cm csi-kms-connection-details -n openshift-storage -o yaml apiVersion: v1 data: vault-token-v1: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault-token-v1","VAULT_ADDR":"https://vault.ocs.com:8200","VAULT_BACKEND_PATH":"vb-sys-v1","VAULT_CACERT":"ocs-kms-ca-secret-izzsps","VAULT_TLS_SERVER_NAME":"","VAULT_CLIENT_CERT":"ocs-kms-client-cert-3jnpp","VAULT_CLIENT_KEY":"ocs-kms-client-key-r1f8op","VAULT_NAMESPACE":"","VAULT_TOKEN_NAME":"","VAULT_CACERT_FILE":"fullchain.pem","VAULT_CLIENT_CERT_FILE":"cert.pem","VAULT_CLIENT_KEY_FILE":"privkey.pem"}' kind: ConfigMap The backend path overridden in the tenant namespace: ---------------------------------------------------- $ oc get cm ceph-csi-kms-config -o yaml apiVersion: v1 data: VAULT_BACKEND_PATH: test-sa kind: ConfigMap The PVC creation fails (see error below) since ceph-csi tries to store the keys in the backend path provided in the csi-kms-connection-details configmap, whereas the backend path and vault token in the tenant namespace points to a different backend path. When the tenant configmap is edited to use camelCase key, the PVC creation succeeds and the the backend path is overridden to use the one specified in the tenant configmap. $ oc get cm ceph-csi-kms-config -o yaml apiVersion: v1 data: vaultBackendPath: test-sa kind: ConfigMap $ oc describe pvc pvc-rbd-block-rwo-1 [...] Warning ProvisioningFailed 11m openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5c85747995-4z9f5_61dba468-9feb-4785-bf62-4afa4d2874df (combined from similar events): failed to provision volume with StorageClass "test-pv-encryption-1": rpc error: code = Internal desc = failed to setup encryption for image ocs-storagecluster-cephblockpool/csi-vol-4551c3ed-19e9-11ec-9980-0a580a830024: failed to save the passphrase for 0001-0011-openshift-storage-0000000000000001-4551c3ed-19e9-11ec-9980-0a580a830024: saving passphrase at 0001-0011-openshift-storage-0000000000000001-4551c3ed-19e9-11ec-9980-0a580a830024 request to vault failed: failed to put secret: backendPath=vb-sys-v1, backendV2=false, namespace=, secretID=0001-0011-openshift-storage-0000000000000001-4551c3ed-19e9-11ec-9980-0a580a830024: Error making API request. URL: PUT https://vault.ocs.com:8200/v1/vb-sys-v1/0001-0011-openshift-storage-0000000000000001-4551c3ed-19e9-11ec-9980-0a580a830024 Code: 403. Errors: * 1 error occurred: * permission denied Normal ExternalProvisioning 9m56s (x26 over 15m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "openshift-storage.rbd.csi.ceph.com" or manually created by system administrator Normal Provisioning 6m54s (x11 over 15m) openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5c85747995-4z9f5_61dba468-9feb-4785-bf62-4afa4d2874df External provisioner is provisioning volume for claim "test-1/pvc-rbd-block-rwo-1" Normal ProvisioningSucceeded 6m54s openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5c85747995-4z9f5_61dba468-9feb-4785-bf62-4afa4d2874df Successfully provisioned volume pvc-e232b462-8638-46a9-b3b7-ba8c6c91980c $ vault kv list test-sa Keys ---- 0001-0011-openshift-storage-0000000000000001-ddf958b9-19e9-11ec-9980-0a580a830024 Version of all relevant components (if applicable): =================================================== ODF: odf-operator.v4.9.0-142.ci OCP: 4.9.0-0.nightly-2021-09-18-052905 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? ================================================================ Yes, this issue is not observed if the key in the tenant configmap is specified in camelCase $ oc get cm ceph-csi-kms-config -o yaml apiVersion: v1 data: vaultBackendPath: test-sa kind: ConfigMap The backend path specified in the tenant configmap is used, instead of the path specified in the csi-kms-connection-details configmap Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: No Steps to Reproduce: =================== 1. Create an RBD storageclass with encryption enabled $ oc get cm csi-kms-connection-details -n openshift-storage -o yaml apiVersion: v1 data: vault-token-v1: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault-token-v1","VAULT_ADDR":"https://vault.ocs.com:8200","VAULT_BACKEND_PATH":"vb-sys-v1","VAULT_CACERT":"ocs-kms-ca-secret-izzsps","VAULT_TLS_SERVER_NAME":"","VAULT_CLIENT_CERT":"ocs-kms-client-cert-3jnpp","VAULT_CLIENT_KEY":"ocs-kms-client-key-r1f8op","VAULT_NAMESPACE":"","VAULT_TOKEN_NAME":"","VAULT_CACERT_FILE":"fullchain.pem","VAULT_CLIENT_CERT_FILE":"cert.pem","VAULT_CLIENT_KEY_FILE":"privkey.pem"}' kind: ConfigMap $ oc get sc test-pv-encryption-1 -o yaml allowVolumeExpansion: false apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: creationTimestamp: "2021-09-20T06:48:15Z" name: test-pv-encryption-1 resourceVersion: "48444" uid: 2a27f12f-3834-43f0-a611-2477bcce3fb7 parameters: clusterID: openshift-storage csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage encrypted: "true" encryptionKMSID: vault-token-v1 imageFeatures: layering imageFormat: "2" pool: ocs-storagecluster-cephblockpool provisioner: openshift-storage.rbd.csi.ceph.com reclaimPolicy: Delete volumeBindingMode: Immediate 2. In the tenant namespace create a tenant configmap as shown below, with a different backend path $ oc get cm ceph-csi-kms-config -o yaml apiVersion: v1 data: VAULT_BACKEND_PATH: test-sa kind: ConfigMap metadata: creationTimestamp: "2021-09-20T07:26:25Z" name: ceph-csi-kms-config namespace: test-1 resourceVersion: "69363" uid: 70d437e3-1b7a-4b5c-9ccf-bee14b9d88d9 3. Create a secret in the tenant namespace containing the vault token. 4. Create a PVC using the storageclass created above 5. Check the status of the PVC Actual results: =============== The provisioning of the PVC fails with the following error: Warning ProvisioningFailed 4s openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5c85747995-4z9f5_61dba468-9feb-4785-bf62-4afa4d2874df failed to provision volume with StorageClass "test-pv-encryption-1": rpc error: code = Internal desc = failed to setup encryption for image ocs-storagecluster-cephblockpool/csi-vol-b5f8904f-19e8-11ec-9980-0a580a830024: failed to save the passphrase for 0001-0011-openshift-storage-0000000000000001-b5f8904f-19e8-11ec-9980-0a580a830024: saving passphrase at 0001-0011-openshift-storage-0000000000000001-b5f8904f-19e8-11ec-9980-0a580a830024 request to vault failed: failed to put secret: backendPath=vb-sys-v1, backendV2=false, namespace=, secretID=0001-0011-openshift-storage-0000000000000001-b5f8904f-19e8-11ec-9980-0a580a830024: Error making API request. Ceph-CSI still tries to store the keys in backend path "vb-sys-v1", but the backend path specified in the tenant configmap is "test-sa". Expected results: ================= The PVC should be in bound state, with the key being stored in the backend path specified in the tenant configmap. The tenant configmap should accept both UPPER_CASE and camelCase keys.