Bug 1467244 - Dynamic Volume Provisioner creates unencrypted volumes [NEEDINFO]
Dynamic Volume Provisioner creates unencrypted volumes
Status: ASSIGNED
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.8.0
Unspecified Unspecified
unspecified Severity low
: ---
: 3.7.z
Assigned To: Jan Safranek
chaoyang
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-03 04:45 EDT by Vladislav Walek
Modified: 2018-06-18 00:21 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
suchaudh: needinfo? (jsafrane)


Attachments (Terms of Use)

  None (edit)
Description Vladislav Walek 2017-07-03 04:45:14 EDT
Description of problem:

We are using the AWS dynamic volume provisioner with KMS encrypted EBS volumes. 
If the storageclass is configured with "encrypted: true" an a KMS key id but no key rights are granted for the EC2 instance, the volume creation should fail.
Instead, the volumes are created unencrypted.

Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.5 on AWS

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
Comment 1 Jan Safranek 2017-07-03 08:46:56 EDT
I'll look at it
Comment 2 Jan Safranek 2017-07-03 10:51:01 EDT
I reproduced something, not sure if it is the same bug... When StorageClass for AWS refers to an non-existing encryption key then dynamic provisioning looks like it's working, it provisions a PV and binds it to a PVC, but the underlying AWS EBS does not exist.

The reason is that AWS API returns a successfully created AWS EBS when creating one with invalid KSM:

$ aws ec2 create-volume --size=1 --volume-type=gp2 --availability-zone=us-east-1d --encrypted --kms-key-id=arn:aws:kms:us-east-1:599999999999:alias/nonExistingKey

{
    "VolumeId": "vol-0441169d687c7f108",
    "Size": 1,
...
}

But the volume actually does not exist:

$ aws ec2 describe-volumes --volume-ids=vol-0441169d687c7f108

An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumes operation: The volume 'vol-0441169d687c7f108' does not exist.


Please ask the customer if it observes similar behavior (or what behavior do they actually observe... openshift-master logs would be helpful)
Comment 3 Jan Safranek 2017-07-03 11:16:07 EDT
Filled an issue upstream: https://github.com/kubernetes/kubernetes/issues/48438
Comment 6 Jan Safranek 2017-07-14 07:52:06 EDT
Posted a PR  upstream: https://github.com/kubernetes/kubernetes/pull/48936

Not sure if it fixes customer problem though.
Comment 8 Vladislav Walek 2017-07-20 04:36:41 EDT
Hello Jan,

here are the steps how to reproduce the issue:

this issue is similar to ours, but not the same. Steps to reproduce our issue:

* Create a new KMS key and grant no usage rights to anyone
* Create a new StorageClass with "encrypted: true" and specify the newly created KMS key
* Create a PersistentVolumeClaim

Expected result:
The volume creation should fail, as the OpenShift master does not have the rights to create encrypted EBS volumes with the specified KMS key.

Actual result:
The volume creation succeeds but the volume is not encrypted.
Comment 9 Jan Safranek 2017-07-20 12:00:05 EDT
> The volume creation succeeds but the volume is not encrypted.

I see a different behavior. On my machine, Kubernetes looks like it provisions AWS EBS + Kubernetes PV, however the EBS disappears in a second. I can't check if it's encrypted or unencrypted. I'll fix it by checking if any provided key exists before trying to create a volume with it. If nobody (incl. Kubernetes) has no access to the key, then I can't check existence of the key and it will behave the same way...
Comment 12 Jan Safranek 2017-09-08 06:59:50 EDT
Origin PR: https://github.com/openshift/origin/pull/16234
Comment 16 Jan Safranek 2018-01-08 04:58:11 EST
It should be fixed in 3.8 by Kubernetes rebase.
Comment 27 Jan Safranek 2018-04-03 09:05:30 EDT
I can confirm the bug persists if OpenShift gets AccessDeniedException when checking KMS key.

And this can't be easily fixed: AccessDeniedException says that OpenShift can't access the key, it does not say that OpenShift can't provision volumes with that key - KMS may allow AWS EBS to use the key for encryption.

AWS does *not* return any sane error code when provisioning with inaccessible key fails, it returns a volume that exists for a while and then it's deleted.

AWS suggests is to subscribe to CloudWatch Events:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-cloud-watch-events.html#create-fail-key

This will require 1) quite big change in AWS provider and 2) OpenShift needs to get new permissions to read CloudWatch, which is IMO no-go.

Note You need to log in before you can comment on or make changes to this bug.