Red Hat Bugzilla – Bug 1467244
Dynamic Volume Provisioner creates unencrypted volumes
Last modified: 2018-06-18 00:21:10 EDT
Description of problem:
We are using the AWS dynamic volume provisioner with KMS encrypted EBS volumes.
If the storageclass is configured with "encrypted: true" an a KMS key id but no key rights are granted for the EC2 instance, the volume creation should fail.
Instead, the volumes are created unencrypted.
Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.5 on AWS
Steps to Reproduce:
Node Log (of failed PODs):
StorageClass Dump (if StorageClass used by PV/PVC):
I'll look at it
I reproduced something, not sure if it is the same bug... When StorageClass for AWS refers to an non-existing encryption key then dynamic provisioning looks like it's working, it provisions a PV and binds it to a PVC, but the underlying AWS EBS does not exist.
The reason is that AWS API returns a successfully created AWS EBS when creating one with invalid KSM:
$ aws ec2 create-volume --size=1 --volume-type=gp2 --availability-zone=us-east-1d --encrypted --kms-key-id=arn:aws:kms:us-east-1:599999999999:alias/nonExistingKey
But the volume actually does not exist:
$ aws ec2 describe-volumes --volume-ids=vol-0441169d687c7f108
An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumes operation: The volume 'vol-0441169d687c7f108' does not exist.
Please ask the customer if it observes similar behavior (or what behavior do they actually observe... openshift-master logs would be helpful)
Filled an issue upstream: https://github.com/kubernetes/kubernetes/issues/48438
Posted a PR upstream: https://github.com/kubernetes/kubernetes/pull/48936
Not sure if it fixes customer problem though.
here are the steps how to reproduce the issue:
this issue is similar to ours, but not the same. Steps to reproduce our issue:
* Create a new KMS key and grant no usage rights to anyone
* Create a new StorageClass with "encrypted: true" and specify the newly created KMS key
* Create a PersistentVolumeClaim
The volume creation should fail, as the OpenShift master does not have the rights to create encrypted EBS volumes with the specified KMS key.
The volume creation succeeds but the volume is not encrypted.
> The volume creation succeeds but the volume is not encrypted.
I see a different behavior. On my machine, Kubernetes looks like it provisions AWS EBS + Kubernetes PV, however the EBS disappears in a second. I can't check if it's encrypted or unencrypted. I'll fix it by checking if any provided key exists before trying to create a volume with it. If nobody (incl. Kubernetes) has no access to the key, then I can't check existence of the key and it will behave the same way...
Origin PR: https://github.com/openshift/origin/pull/16234
It should be fixed in 3.8 by Kubernetes rebase.
I can confirm the bug persists if OpenShift gets AccessDeniedException when checking KMS key.
And this can't be easily fixed: AccessDeniedException says that OpenShift can't access the key, it does not say that OpenShift can't provision volumes with that key - KMS may allow AWS EBS to use the key for encryption.
AWS does *not* return any sane error code when provisioning with inaccessible key fails, it returns a volume that exists for a while and then it's deleted.
AWS suggests is to subscribe to CloudWatch Events:
This will require 1) quite big change in AWS provider and 2) OpenShift needs to get new permissions to read CloudWatch, which is IMO no-go.