Friday we had a bomgar in which the customer applied the proposed cinder fix in their rhos6 environment. It worked in that nova now attempts luks encryption instead of acting as if the volume was unencrypted, but the encryption commands failed: 2016-07-29 14:39:13.061 7007 DEBUG nova.volume.encryptors.luks [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] opening encrypted volume /dev/dm-8 _open_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:66 2016-07-29 14:39:13.062 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634756 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161 2016-07-29 14:39:13.745 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Result was 1 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195 2016-07-29 14:39:13.746 7007 DEBUG nova.volume.encryptors.luks [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] formatting encrypted volume /dev/dm-8 _format_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:41 2016-07-29 14:39:13.746 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 512 /dev/dm-8 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161 2016-07-29 14:39:17.676 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Result was 0 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195 2016-07-29 14:39:17.677 7007 DEBUG nova.volume.encryptors.luks [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] opening encrypted volume /dev/dm-8 _open_volume /usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py:66 2016-07-29 14:39:17.677 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634756 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161 2016-07-29 14:39:17.751 7007 DEBUG nova.openstack.common.processutils [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] Result was 1 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195 2016-07-29 14:39:17.751 7007 ERROR nova.virt.libvirt.driver [req-c613fa13-104f-4e47-9fe0-b87857ca35ed None] [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Failed to attach volume at mountpoint: /dev/vdb 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Traceback (most recent call last): 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1413, in attach_volume 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] encryptor.attach_volume(context, **encryption) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py", line 92, in attach_volume 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] self._open_volume(passphrase, **kwargs) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/luks.py", line 69, in _open_volume 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] run_as_root=True, check_exit_code=True) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 187, in execute 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] return processutils.execute(*cmd, **kwargs) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 203, in execute 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] cmd=sanitized_cmd) 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] ProcessExecutionError: Unexpected error while running command. 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/dm-8 3600a098038303365763f476c63634756 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Exit code: 1 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Stdout: u'' 2016-07-29 14:39:17.751 7007 TRACE nova.virt.libvirt.driver [instance: dd0b3e29-f8c5-4d66-b96c-50c25709ead6] Stderr: u'' The first luksOpen failure, which just returns 1 and does not produce an exception, is expected because the volume is not yet luksFormatted. The cryptsetup command that follows to do the luksFormat succeeds with return code 0. However, the following luksOpen again fails, unexpectedly, and causes an attach volume exception. I have since been table to test with rhos, the proposed cinder patch, and a NetApp iscsi backend. Volume encryption is working: encrypted volumes attach successfully to compute instances, their contents appear as plaintext from the vantage of the compute instance, and their contents are luks encrypted from the vantage of the nova host. I suspect that the issue *may* go away if the customer upgrades his nova rpm. He's using openstack-nova-compute-2014.2.3-9.el7ost.noarch where as I was at openstack-nova-compute-2014.2.3-74.el7ost.noarch. That said, I have proposed a backport of a fix from Liberty that retries the luksFormat operation in case the device since the sequence of commands here appears to be sensitive to timing. I think we should have the customer update his nova packages to the latest plus this fix when it is built and try the encryption test again. Since his failure was with the second open and the format operation apparently succeeded, it is possible that we will need to implement a retry loop for the second open as well.
Tested using: openstack-cinder-2015.1.3-8.el7ost.noarch python-cinderclient-1.2.1-3.el7ost.noarch python-cinder-2015.1.3-8.el7ost.noarch Verification flow: [root@cougar01 tempest(keystone_admin)]# cinder type-create LUKS +--------------------------------------+------+ | ID | Name | +--------------------------------------+------+ | 9450e004-7163-4f78-8784-13575c28e2bd | LUKS | +--------------------------------------+------+ [root@cougar01 tempest(keystone_admin)]# cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 \ > --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ | Volume Type ID | Provider | Cipher | Key Size | Control Location | +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ | 9450e004-7163-4f78-8784-13575c28e2bd | nova.volume.encryptors.luks.LuksEncryptor | aes-xts-plain64 | 512 | front-end | +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ [root@cougar01 tempest(keystone_admin)]# cinder type-key LUKS set volume_backend_name=Netapp1 [root@cougar01 tempest(keystone_admin)]# cinder extra-specs-list +--------------------------------------+-------+--------------------------------------+ | ID | Name | extra_specs | +--------------------------------------+-------+--------------------------------------+ | 9450e004-7163-4f78-8784-13575c28e2bd | LUKS | {u'volume_backend_name': u'Netapp1'} | | a46b6b92-fb46-4463-aec2-9b4f7f29484d | iscsi | {u'volume_backend_name': u'lvm'} | +--------------------------------------+-------+--------------------------------------+ [root@cougar01 tempest(keystone_admin)]# cinder create 1 --volume-type LUKS +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2016-08-08T14:48:47.479552 | | display_description | None | | display_name | None | | encrypted | True | | id | c293547a-9f74-4a4d-bbb4-eb5341086239 | | metadata | {} | | multiattach | false | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | LUKS | +---------------------+--------------------------------------+ [root@cougar01 tempest(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | c293547a-9f74-4a4d-bbb4-eb5341086239 | available | - | 1 | LUKS | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [root@cougar01 tempest(keystone_admin)]# nova boot --flavor 1 --image 7513c5ab-2574-4245-9dde-f06d947b21de --nic net-id=a4143383-2383-4fb8-b49a-208debf61cc4 vm +--------------------------------------+-----------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | instance-00000003 | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | MsSzu8Nr5izM | | config_drive | | | created | 2016-08-08T14:47:41Z | | flavor | m1.tiny (1) | | hostId | | | id | 951a6974-0aa8-4dd0-81c7-9acdad8925b4 | | image | cirros (7513c5ab-2574-4245-9dde-f06d947b21de) | | key_name | - | | metadata | {} | | name | vm | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | a593ed86d05e433d8c8b86c8679227f1 | | updated | 2016-08-08T14:47:41Z | | user_id | 97ee77d442434c7583b0cd651a8a8857 | +--------------------------------------+-----------------------------------------------+ [root@cougar01 tempest(keystone_admin)]# nova list +--------------------------------------+------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+---------------------+ | 951a6974-0aa8-4dd0-81c7-9acdad8925b4 | vm | ACTIVE | - | Running | public=172.24.4.230 | +--------------------------------------+------+--------+------------+-------------+---------------------+ [root@cougar01 tempest(keystone_admin)]# nova volume-attach 951a6974-0aa8-4dd0-81c7-9acdad8925b4 c293547a-9f74-4a4d-bbb4-eb5341086239 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c293547a-9f74-4a4d-bbb4-eb5341086239 | | serverId | 951a6974-0aa8-4dd0-81c7-9acdad8925b4 | | volumeId | c293547a-9f74-4a4d-bbb4-eb5341086239 | +----------+--------------------------------------+ [root@cougar01 tempest(keystone_admin)]# cinder list +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+ | c293547a-9f74-4a4d-bbb4-eb5341086239 | in-use | - | 1 | LUKS | false | 951a6974-0aa8-4dd0-81c7-9acdad8925b4 | +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1791.html