Description of problem: We tried setting NFS encryption in the past as outlined in http://docs.openstack.org/juno/config-reference/content/section_initial-configuration.html and tried to perfrom the validation at http://docs.openstack.org/juno/config-reference/content/section_testing_encryption.html which was covered on case 01569966. As this failed, we went to engineering and we were told that file based backends or 'rbd' where not supported: "The volume encryption in Nova was only ever designed to work with block device based volumes. Support for network attached volumes (RBD) or file based volumes (NFS) is a future RFE upstream, pending on QEMU support for LUKS. So the report is testing a feature which is known to not exist at this time. As such I'm marking this an RFE, since its not a bug." (from https://bugzilla.redhat.com/show_bug.cgi?id=1305024#c2 ) So we configured iSCSI with Netapp Driver and after performing the same tests as described in upstream guide (write from within instance to the encrypted volume and use 'strings' from compute on the raw disk file), we still can see the strings in clear-text Both cinder and nova conf files contain the encryption key (static) as defined in the configuration
Friday we had a bomgar in which the customer applied the proposed cinder fix in their rhos6 environment. It worked in that nova now attempts luks encryption instead of acting as if the volume was unencrypted, but the encryption commands failed: 2016-07-29 14:39:13.061 7007 DEBUG nova.volume.encryptors.luks [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] opening encrypted volume /dev/dm-8 _open_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:66 2016-07-29 14:39:13.062 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634756 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161 2016-07-29 14:39:13.745 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Result was 1 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195 2016-07-29 14:39:13.746 7007 DEBUG nova.volume.encryptors.luks [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] formatting encrypted volume /dev/dm-8 _format_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:41 2016-07-29 14:39:13.746 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 512 /dev/dm-8 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161 2016-07-29 14:39:17.676 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Result was 0 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195 2016-07-29 14:39:17.677 7007 DEBUG nova.volume.encryptors.luks [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] opening encrypted volume /dev/dm-8 _open_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:66 2016-07-29 14:39:17.677 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634756 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161 2016-07-29 14:39:17.751 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Result was 1 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195 2016-07-29 14:39:17.751 7007 ERROR nova.virt.libvirt.driver [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Failed to attach volume at mountpoint: /dev/vdb 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Traceback (most recent call last): 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1413, in attach_volume 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] encryptor.attach_volume(context, **encryption) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py", line 92, in attach_volume 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] self._open_volume(passphrase, **kwargs) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py", line 69, in _open_volume 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] run_as_root=True, check_exit_code=True) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 187, in execute 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] return processutils.execute(*cmd, **kwargs) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 203, in execute 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] cmd=sanitized_cmd) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] ProcessExecutionError: Unexpected error while running command. 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634756 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Exit code: 1 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Stdout: u'' 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Stderr: u'' The first luksOpen failure, which just returns 1 and does not produce an exception, is expected because the volume is not yet luksFormatted. The cryptsetup command that follows to do the luksFormat succeeds with return code 0. However, the following luksOpen again fails, unexpectedly, and causes an attach volume exception. I have since been table to test with rhos, the proposed cinder patch, and a NetApp iscsi backend. Volume encryption is working: encrypted volumes attach successfully to compute instances, their contents appear as plaintext from the vantage of the compute instance, and their contents are luks encrypted from the vantage of the nova host. I suspect that the issue *may* go away if the customer upgrades his nova rpm. He's using openstack-nova-compute-2014.2.3-9.el7ost.noarch where as I was at openstack-nova-compute-2014.2.3-74.el7ost.noarch. That said, I have proposed a backport of a fix from Liberty that retries the luksFormat operation in case the device since the sequence of commands here appears to be sensitive to timing. I think we should have the customer update his nova packages to the latest plus this fix when it is built and try the encryption test again. Since his failure was with the second open and the format operation apparently succeeded, it is possible that we will need to implement a retry loop for the second open as well.
Hi Tom, it also failed with the latest nova : 2016-08-01 18:22:25.241 13629 AUDIT nova.service [-] Starting compute node (version 2014.2.3-74.el7ost) ... 2016-08-01 18:27:27.486 13629 DEBUG nova.virt.libvirt.volume [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] multipath ['-r']: stdout=reload: 3600a098038303365763f476c63634659 undef NETAPP ,LUN C-Mode size=30G features='3 queue_if_no_path pg_init_retries 50' hwhandler='0' wp=undef |-+- policy='service-time 0' prio=50 status=undef | |- 0:0:1:0 sdb 8:16 active ready running | `- 1:0:1:0 sdd 8:48 active ready running `-+- policy='service-time 0' prio=10 status=undef |- 0:0:0:0 sda 8:0 active ready running `- 1:0:0:0 sdc 8:32 active ready running reload: 3600a098038303365763f476c63634758 undef NETAPP ,LUN C-Mode size=1.0G features='3 queue_if_no_path pg_init_retries 50' hwhandler='0' wp=undef |-+- policy='service-time 0' prio=50 status=undef | |- 37:0:0:0 sdj 8:144 active ready running | `- 38:0:0:0 sdk 8:160 active ready running `-+- policy='service-time 0' prio=10 status=undef |- 36:0:0:0 sdi 8:128 active ready running `- 39:0:0:0 sdl 8:176 active ready running stderr= _run_multipath /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py:679 2016-08-01 18:27:27.486 13629 DEBUG nova.openstack.common.processutils [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 ] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l /dev/sdi execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils. py:171 ... 2016-08-01 18:27:27.548 13629 WARNING nova.keymgr.single_key_mgr [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] This key manager is insecure and is not recommended for production deployments 2016-08-01 18:27:27.548 13629 WARNING nova.keymgr.mock_key_mgr [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] This key manager is not suitable for use in production deployments 2016-08-01 18:27:27.549 13629 WARNING nova.keymgr.single_key_mgr [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] This key manager is insecure and is not recommended for production deployments 2016-08-01 18:27:27.549 13629 WARNING nova.keymgr.mock_key_mgr [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] This key manager is not suitable for use in production deployments 2016-08-01 18:27:27.549 13629 DEBUG nova.volume.encryptors.luks [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] opening encrypted volume /dev/dm-8 _open_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:66 2016-08-01 18:27:27.550 13629 DEBUG nova.openstack.common.processutils [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 ] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634758 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:171 2016-08-01 18:27:27.628 13629 DEBUG nova.openstack.common.processutils [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 ] Result was 1 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:210 2016-08-01 18:27:27.629 13629 DEBUG nova.volume.encryptors.luks [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] formatting encrypted volume /dev/dm-8 _format_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:41 2016-08-01 18:27:27.629 13629 DEBUG nova.openstack.common.processutils [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 ] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 512 /dev/dm-8 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:171 ... 2016-08-01 18:27:32.081 13629 DEBUG nova.volume.encryptors.luks [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] opening encrypted volume /dev/dm-8 _open_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:66 2016-08-01 18:27:32.081 13629 DEBUG nova.openstack.common.processutils [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 ] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634758 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:171 2016-08-01 18:27:32.160 13629 DEBUG nova.openstack.common.processutils [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 ] Result was 1 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:210 2016-08-01 18:27:32.161 13629 ERROR nova.virt.libvirt.driver [req-945b0e6f-b1ed-47b1-9fec-86058d8a2225 None] [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] Failed to attach volume at mountpoint: /dev/vdb 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] Traceback (most recent call last): 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1479, in attach_volume 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] encryptor.attach_volume(context, **encryption) 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py", line 92, in attach_volume 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] self._open_volume(passphrase, **kwargs) 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py", line 69, in _open_volume 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] run_as_root=True, check_exit_code=True) 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 207, in execute 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] return processutils.execute(*cmd, **kwargs) 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 222, in execute 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] cmd=sanitized_cmd) 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] ProcessExecutionError: Unexpected error while running command. 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634758 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] Exit code: 1 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] Stdout: u'' 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823] Stderr: u'' 2016-08-01 18:27:32.161 13629 TRACE nova.virt.libvirt.driver [instance: 35642e53-9ba8-4899-a767-2b58fe6fb823]
Tested using: python-cinderclient-1.1.1-3.el7ost.noarch python-cinder-2014.2.4-8.el7ost.noarch openstack-cinder-2014.2.4-8.el7ost.noarch Verification flow: [root@cougar01 ~(keystone_admin)]# cinder type-create LUKS +--------------------------------------+------+ | ID | Name | +--------------------------------------+------+ | 42a17f69-98eb-413f-a671-f80680674bb3 | LUKS | +--------------------------------------+------+ [root@cougar01 ~(keystone_admin)]# cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 \ > --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ | Volume Type ID | Provider | Cipher | Key Size | Control Location | +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ | 42a17f69-98eb-413f-a671-f80680674bb3 | nova.volume.encryptors.luks.LuksEncryptor | aes-xts-plain64 | 512 | front-end | +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ [root@cougar01 ~(keystone_admin)]# cinder type-key LUKS set volume_backend_name=Netapp1 [root@cougar01 ~(keystone_admin)]# cinder extra-specs-list +--------------------------------------+-------+--------------------------------------+ | ID | Name | extra_specs | +--------------------------------------+-------+--------------------------------------+ | 42a17f69-98eb-413f-a671-f80680674bb3 | LUKS | {u'volume_backend_name': u'Netapp1'} | | 7607be83-5ccc-4a69-9beb-bb212186fa22 | iscsi | {u'volume_backend_name': u'lvm'} | +--------------------------------------+-------+--------------------------------------+ [root@cougar01 ~(keystone_admin)]# cinder create 1 --volume-type LUKS +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2016-08-08T07:13:42.484900 | | display_description | None | | display_name | None | | encrypted | True | | id | ce9ec4bf-f793-431c-8bb5-d247bae836b9 | | metadata | {} | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | LUKS | +---------------------+--------------------------------------+ [root@cougar01 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ce9ec4bf-f793-431c-8bb5-d247bae836b9 | available | None | 1 | LUKS | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [root@cougar01 ~(keystone_admin)]# cinder show ce9ec4bf-f793-431c-8bb5-d247bae836b9 +---------------------------------------+----------------------------------------------+ | Property | Value | +---------------------------------------+----------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2016-08-08T07:13:42.000000 | | display_description | None | | display_name | None | | encrypted | True | | id | ce9ec4bf-f793-431c-8bb5-d247bae836b9 | | metadata | {} | | os-vol-host-attr:host | cougar01.scl.lab.tlv.redhat.com@Netapp1#RHEV | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | c87cc8da802e4ed4adccf4650fd01e2f | | os-volume-replication:driver_data | None | | os-volume-replication:extended_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | volume_type | LUKS | +---------------------------------------+----------------------------------------------+
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1618.html