Bug 1983931

Eran/Niels, are we tagetting this for 4.7.4 or 4.7.3?

From an engineering point of view, we can include this in 4.7.3. Currently a PR is available and waiting for the opening of the 4.7.4 merge window.

The customer that reported the problem, can apply a temporary workaround if needed.

If other stakeholders are supporting to include this in 4.7.3, we can include the change in the next build for testing.

Eran clarified that this is needed in 4.8.1 also, so we need to target this to 4.7.4
Will create a 4.8.1 clone too

The only valid values for `VAULT_BACKEND` are `kv` and `kv-v2`. If the option is not set, or has an other value, libopenstorage/secrets will try to do auto-detection. Only when auto-detection fails (no access to `sys/mounts`), connecting to the Vault service will fail.

`kv-v2` formatting of the URLs is different than the format for `kv` (version 1)`:

- kv: no support for metadata, no need to separate data
- kv-v2: supports metadata, different URLs for data and metadata

When using a `kv` (version 1) storage, the secret store does not get a `data/` part in the name of the key. When using `kv-v2`, the name of the key will get an additional `data/` section in the path.

The following configuration has now been tested, and it should show that the setting of `VAULT_BACKEND` is used:

- configure a `vaulttokens` KMS
- create a `secret/` store, with `kv -version=1`
- create a policy for a tenant, with only access to the `secret/` path, not `sys/mounts`
- create a new token (with -orphan and -policy=...)
- configure the token in the Secret ceph-csi-kms-token in the Tenants namespace

When not setting `VAULT_BACKEND` in the KMS configuration, the provisioning fails with errors like:

--- %< ---
E0806 11:45:20.005724       1 utils.go:136] ID: 138 Req-ID: pvc-f6c890dc-dce3-47cb-8c24-8b6d79c2373b GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed creating new Vault Secrets: Error making API request.

URL: GET http://vault.infra.svc.cluster.local:8200/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
	* permission denied

--- %< ---


Setting `VAULT_BACKEND` to "invalid" causes the same error (auto-detection fails to access "sys/mounts"):

--- %< ---
E0806 11:47:24.817460       1 utils.go:136] ID: 148 Req-ID: pvc-9e70ce37-6b00-4032-a8e4-835d833a372b GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed creating new Vault Secrets: Error making API request.

URL: GET http://vault.infra.svc.cluster.local:8200/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
	* permission denied

--- %< ---


Setting `VAULT_BACKEND` to "kv" (version 1) and creating a PVC and Pod results in a key created in Vault:

--- %< ---
/ $ vault kv list secret/
Keys
----
0001-0011-openshift-storage-0000000000000001-48698aff-f6ac-11eb-9b5d-0a580a81025a
/ $ vault kv get secret/0001-0011-openshift-storage-0000000000000001-48698aff-f6ac-11eb-9b5d-0a580a81025a
==== Data ====
Key     Value
---     -----
data    map[passphrase:QD9Tph39Hl1LYTUJg9Yoc-SwcoY=]
--- %< ---


Setting `VAULT_BACKEND` to "kv-v2" (version 2) and creating a PVC and Pod results in a key created in Vault, but added "data/" path (required for version 2):

--- %< ---
/ $ vault kv list secret/
Keys
----
data/
/ $ vault kv list secret/data
Keys
----
0001-0011-openshift-storage-0000000000000001-cf18f1a7-f6ac-11eb-9b5d-0a580a81025a
/ $ vault kv get secret/data/0001-0011-openshift-storage-0000000000000001-cf18f1a7-f6ac-11eb-9b5d-0a580a81025a
==== Data ====
Key     Value
---     -----
data    map[data:map[passphrase:fqHbBr7LRQVYIXvO9L0mtO8eb9M=]]
--- %< ---


According to my understanding, this confirms that the `VAULT_BACKEND` configuration option in the csi-kms-connection-details ConfigMap has an effect on how the encryption passphrase is stored in Vault.

Also note that validating `kv` (version 1) and `kv-v2` (version 2) can be done by using policies that are restrictive:

For `kv`:
 - allow writing to `secret/*`, this will also work when configuring with `kv-v2` as it uses sub-paths which are allowed automatically
 - maybe it is possible to deny access to `secret/data/*`, in which case using `kv-v2` should fail

For `kv-v2`:
 - allow writing to only `secret/data/*` and secret/metadata/*`, this will prevent using `kv` as it tries to use `secret/*`

Moving it back to ON_QA based on the above comments.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Container Storage 4.7.3 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3135