Bug 1467244

I'll look at it

I reproduced something, not sure if it is the same bug... When StorageClass for AWS refers to an non-existing encryption key then dynamic provisioning looks like it's working, it provisions a PV and binds it to a PVC, but the underlying AWS EBS does not exist.

The reason is that AWS API returns a successfully created AWS EBS when creating one with invalid KSM:

$ aws ec2 create-volume --size=1 --volume-type=gp2 --availability-zone=us-east-1d --encrypted --kms-key-id=arn:aws:kms:us-east-1:599999999999:alias/nonExistingKey

{
    "VolumeId": "vol-0441169d687c7f108",
    "Size": 1,
...
}

But the volume actually does not exist:

$ aws ec2 describe-volumes --volume-ids=vol-0441169d687c7f108

An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumes operation: The volume 'vol-0441169d687c7f108' does not exist.


Please ask the customer if it observes similar behavior (or what behavior do they actually observe... openshift-master logs would be helpful)

Filled an issue upstream: https://github.com/kubernetes/kubernetes/issues/48438

Posted a PR  upstream: https://github.com/kubernetes/kubernetes/pull/48936

Not sure if it fixes customer problem though.

Hello Jan,

here are the steps how to reproduce the issue:

this issue is similar to ours, but not the same. Steps to reproduce our issue:

* Create a new KMS key and grant no usage rights to anyone
* Create a new StorageClass with "encrypted: true" and specify the newly created KMS key
* Create a PersistentVolumeClaim

Expected result:
The volume creation should fail, as the OpenShift master does not have the rights to create encrypted EBS volumes with the specified KMS key.

Actual result:
The volume creation succeeds but the volume is not encrypted.

> The volume creation succeeds but the volume is not encrypted.

I see a different behavior. On my machine, Kubernetes looks like it provisions AWS EBS + Kubernetes PV, however the EBS disappears in a second. I can't check if it's encrypted or unencrypted. I'll fix it by checking if any provided key exists before trying to create a volume with it. If nobody (incl. Kubernetes) has no access to the key, then I can't check existence of the key and it will behave the same way...

Origin PR: https://github.com/openshift/origin/pull/16234

It should be fixed in 3.8 by Kubernetes rebase.

I can confirm the bug persists if OpenShift gets AccessDeniedException when checking KMS key.

And this can't be easily fixed: AccessDeniedException says that OpenShift can't access the key, it does not say that OpenShift can't provision volumes with that key - KMS may allow AWS EBS to use the key for encryption.

AWS does *not* return any sane error code when provisioning with inaccessible key fails, it returns a volume that exists for a while and then it's deleted.

AWS suggests is to subscribe to CloudWatch Events:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-cloud-watch-events.html#create-fail-key

This will require 1) quite big change in AWS provider and 2) OpenShift needs to get new permissions to read CloudWatch, which is IMO no-go.

Origin PR: https://github.com/openshift/origin/pull/20072

It's fixed in v3.10.8-1, waiting for qa to pick it up.

Tried the same step as in #comment 22,

# oc describe pvc mypvc
Name:          mypvc
Namespace:     default
StorageClass:  test
Status:        Pending
Volume:        
Labels:        name=dynamic-pvc
Annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/aws-ebs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  3m (x26 over 9m)  persistentvolume-controller  Failed to provision volume with StorageClass "test": failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key



According #comment 32, we have clear error message now. 
Moving bug to verified.


# oc version
oc v3.10.34
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 (Maipo)

# uname -a
Linux ip-172-18-8-164.ec2.internal 3.10.0-862.9.1.el7.x86_64 #1 SMP Wed Jun 27 04:30:39 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2376