Description of problem: We are using the AWS dynamic volume provisioner with KMS encrypted EBS volumes. If the storageclass is configured with "encrypted: true" an a KMS key id but no key rights are granted for the EC2 instance, the volume creation should fail. Instead, the volumes are created unencrypted. Version-Release number of selected component (if applicable): OpenShift Container Platform 3.5 on AWS How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
I'll look at it
I reproduced something, not sure if it is the same bug... When StorageClass for AWS refers to an non-existing encryption key then dynamic provisioning looks like it's working, it provisions a PV and binds it to a PVC, but the underlying AWS EBS does not exist. The reason is that AWS API returns a successfully created AWS EBS when creating one with invalid KSM: $ aws ec2 create-volume --size=1 --volume-type=gp2 --availability-zone=us-east-1d --encrypted --kms-key-id=arn:aws:kms:us-east-1:599999999999:alias/nonExistingKey { "VolumeId": "vol-0441169d687c7f108", "Size": 1, ... } But the volume actually does not exist: $ aws ec2 describe-volumes --volume-ids=vol-0441169d687c7f108 An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumes operation: The volume 'vol-0441169d687c7f108' does not exist. Please ask the customer if it observes similar behavior (or what behavior do they actually observe... openshift-master logs would be helpful)
Filled an issue upstream: https://github.com/kubernetes/kubernetes/issues/48438
Posted a PR upstream: https://github.com/kubernetes/kubernetes/pull/48936 Not sure if it fixes customer problem though.
Hello Jan, here are the steps how to reproduce the issue: this issue is similar to ours, but not the same. Steps to reproduce our issue: * Create a new KMS key and grant no usage rights to anyone * Create a new StorageClass with "encrypted: true" and specify the newly created KMS key * Create a PersistentVolumeClaim Expected result: The volume creation should fail, as the OpenShift master does not have the rights to create encrypted EBS volumes with the specified KMS key. Actual result: The volume creation succeeds but the volume is not encrypted.
> The volume creation succeeds but the volume is not encrypted. I see a different behavior. On my machine, Kubernetes looks like it provisions AWS EBS + Kubernetes PV, however the EBS disappears in a second. I can't check if it's encrypted or unencrypted. I'll fix it by checking if any provided key exists before trying to create a volume with it. If nobody (incl. Kubernetes) has no access to the key, then I can't check existence of the key and it will behave the same way...
Origin PR: https://github.com/openshift/origin/pull/16234
It should be fixed in 3.8 by Kubernetes rebase.
I can confirm the bug persists if OpenShift gets AccessDeniedException when checking KMS key. And this can't be easily fixed: AccessDeniedException says that OpenShift can't access the key, it does not say that OpenShift can't provision volumes with that key - KMS may allow AWS EBS to use the key for encryption. AWS does *not* return any sane error code when provisioning with inaccessible key fails, it returns a volume that exists for a while and then it's deleted. AWS suggests is to subscribe to CloudWatch Events: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-cloud-watch-events.html#create-fail-key This will require 1) quite big change in AWS provider and 2) OpenShift needs to get new permissions to read CloudWatch, which is IMO no-go.
Origin PR: https://github.com/openshift/origin/pull/20072
It's fixed in v3.10.8-1, waiting for qa to pick it up.
Tried the same step as in #comment 22, # oc describe pvc mypvc Name: mypvc Namespace: default StorageClass: test Status: Pending Volume: Labels: name=dynamic-pvc Annotations: volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/aws-ebs Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 3m (x26 over 9m) persistentvolume-controller Failed to provision volume with StorageClass "test": failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key According #comment 32, we have clear error message now. Moving bug to verified. # oc version oc v3.10.34 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO
# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) # uname -a Linux ip-172-18-8-164.ec2.internal 3.10.0-862.9.1.el7.x86_64 #1 SMP Wed Jun 27 04:30:39 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2376