Created attachment 1213817 [details] nova-compute.log Description of problem: packstack / nova / simple keymgr (lvm) setup fails to detach the encrypted volume. Fails on the tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup[compute,id-cbc752ed-b716-4717-910f-956cce965722,image,volume] test. Version-Release number of selected component (if applicable): puppet-nova-9.4.0-1.el7ost.noarch openstack-packstack-puppet-9.0.0-0.4.0rc4.el7ost.noarch python-crypto-2.6.1-1.1.el7.x86_64 libgcrypt-devel-1.5.3-12.el7_1.1.x86_64 openstack-nova-scheduler-14.0.1-3.el7ost.noarch python-novaclient-6.0.0-1.el7ost.noarch openstack-nova-compute-14.0.1-3.el7ost.noarch openstack-nova-common-14.0.1-3.el7ost.noarch openstack-nova-novncproxy-14.0.1-3.el7ost.noarch openstack-nova-cert-14.0.1-3.el7ost.noarch libgcrypt-1.5.3-12.el7_1.1.x86_64 python2-cryptography-1.3.1-3.el7.x86_64 m2crypto-0.21.1-17.el7.x86_64 openstack-packstack-9.0.0-0.4.0rc4.el7ost.noarch lvm2-libs-2.02.166-1.el7.x86_64 openstack-nova-conductor-14.0.1-3.el7ost.noarch openstack-nova-console-14.0.1-3.el7ost.noarch openstack-nova-api-14.0.1-3.el7ost.noarch cryptsetup-libs-1.7.2-1.el7.x86_64 cryptsetup-1.7.2-1.el7.x86_64 lvm2-2.02.166-1.el7.x86_64 python-nova-14.0.1-3.el7ost.noarch How reproducible: frequently Steps to Reproduce: 1. create packstack setup (nothing special) 2. add 64 char key (hexdump -n 32 -v -e '/1 "%02x"' /dev/urandom) to the /etc/nova/nova.conf [key_manager] fixed_key 3. restart the nova compute 4. run tempest test_encrypted_cinder_volumes test (for ex.: ostestr -r test_encrypted_cinder_volumes) Actual results: test failed: failed to reach available status (current in-use) within the required time (300 s). Expected results: test_encrypted_cinder_volumes passes Additional info: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup remove crypt-ip-192.168.1.13:3260-iscsi-iqn.2010-10.org.openstack:volume-9439e922-1051-4d83-87c7-172689ac29da-lun-0 failed according to the nova-compute.log . remove ioctl on crypt-ip-192.168.1.13:3260-iscsi-iqn.2010-10.org.openstack:volume-9439e922-1051-4d83-87c7-172689ac29da-lun-0 failed: Device or resource busy The issue can be lvm/libvirt (related service unit) configuration issue as well, but it is also possible the nova has to wait a little before it can safely use `cryptsetup remove`. libvirt likely asked for removing the disk before `cryptsetup remove` part, it just not completed.
This landed in master before the break so I'm reusing this bug to track it into stable/newton and OSP 10.
upstream/stable/newton patch just landed 2017-01-25
Verified as follows, ******** VERSION ******** [heat-admin@controller-0 ~]$ yum list installed | grep openstack-nova openstack-nova-api.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed openstack-nova-cert.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed openstack-nova-common.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed openstack-nova-compute.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed openstack-nova-conductor.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed openstack-nova-console.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed openstack-nova-novncproxy.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed openstack-nova-scheduler.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed ******* LOGS ******* [heat-admin@controller-0 ~]$ cinder type-create LUKS +--------------------------------------+------+-------------+-----------+ | ID | Name | Description | Is_Public | +--------------------------------------+------+-------------+-----------+ | 5b0b0556-47fe-46f3-b645-4631840dc49c | LUKS | - | True | +--------------------------------------+------+-------------+-----------+ [heat-admin@controller-0 ~]$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 \ > --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ | Volume Type ID | Provider | Cipher | Key Size | Control Location | +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ | 5b0b0556-47fe-46f3-b645-4631840dc49c | nova.volume.encryptors.luks.LuksEncryptor | aes-xts-plain64 | 512 | front-end | +--------------------------------------+-------------------------------------------+-----------------+----------+------------------+ [heat-admin@controller-0 ~]$ cinder create --display-name 'encrypted volume' --volume-type LUKS 1 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2017-02-15T20:16:35.000000 | | description | None | | encrypted | True | | id | 33737407-10a6-4e56-bcf5-666205d82c0c | | metadata | {} | | migration_status | None | | multiattach | False | | name | encrypted volume | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 2fbbb659cb554fb3adffbdb2a127499f | | replication_status | disabled | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | 200007ec0598452c8d02fcf829a42850 | | volume_type | LUKS | +--------------------------------+--------------------------------------+ [heat-admin@controller-0 ~]$ cinder list +--------------------------------------+-----------+------------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------+------+-------------+----------+-------------+ | 33737407-10a6-4e56-bcf5-666205d82c0c | available | encrypted volume | 1 | LUKS | false | | +--------------------------------------+-----------+------------------+------+-------------+----------+-------------+ [heat-admin@controller-0 ~]$ nova list +--------------------------------------+------+--------+------------+-------------+-------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+-------------------+ | f27ce3b7-ec48-4b89-aeda-7980325347de | vm | ACTIVE | - | Running | public=10.0.0.215 | +--------------------------------------+------+--------+------------+-------------+-------------------+ [heat-admin@controller-0 ~]$ [heat-admin@controller-0 ~]$ sudo grep "fixed_key" /etc/nova/nova.conf fixed_key=8b9aacd510dcb09fdaacf684b22e9eec9d199c45e8ff1e75e8541c733f5fbbe3 [heat-admin@controller-0 ~]$ [heat-admin@controller-0 ~]$ sudo service openstack-nova-api restart Redirecting to /bin/systemctl restart openstack-nova-api.service [heat-admin@controller-0 ~]$ sudo service openstack-nova-cert restart Redirecting to /bin/systemctl restart openstack-nova-cert.service [heat-admin@controller-0 ~]$ sudo service openstack-nova-consoleauth restart Redirecting to /bin/systemctl restart openstack-nova-consoleauth.service [heat-admin@controller-0 ~]$ sudo service openstack-nova-scheduler restart Redirecting to /bin/systemctl restart openstack-nova-scheduler.service [heat-admin@controller-0 ~]$ sudo service openstack-nova-conductor restart Redirecting to /bin/systemctl restart openstack-nova-conductor.service [heat-admin@controller-0 ~]$ sudo service openstack-nova-novncproxy restart Redirecting to /bin/systemctl restart openstack-nova-novncproxy.service [heat-admin@controller-0 ~]$ [heat-admin@compute-0 ~]$ sudo grep fixed_key /etc/nova/nova.conf fixed_key=8b9aacd510dcb09fdaacf684b22e9eec9d199c45e8ff1e75e8541c733f5fbbe3 [heat-admin@compute-0 ~]$ sudo service openstack-nova-compute restart Redirecting to /bin/systemctl restart openstack-nova-compute.service [heat-admin@compute-0 ~]$ [heat-admin@controller-0 ~]$ [heat-admin@controller-0 ~]$ nova volume-attach vm 33737407-10a6-4e56-bcf5-666205d82c0c +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | 33737407-10a6-4e56-bcf5-666205d82c0c | | serverId | f27ce3b7-ec48-4b89-aeda-7980325347de | | volumeId | 33737407-10a6-4e56-bcf5-666205d82c0c | +----------+--------------------------------------+ [heat-admin@controller-0 ~]$ cinder list +--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+ | 33737407-10a6-4e56-bcf5-666205d82c0c | in-use | encrypted volume | 1 | LUKS | false | f27ce3b7-ec48-4b89-aeda-7980325347de | +--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+ [heat-admin@controller-0 ~]$ [heat-admin@controller-0 ~]$ nova volume-detach vm 33737407-10a6-4e56-bcf5-666205d82c0c [heat-admin@controller-0 ~]$ [heat-admin@controller-0 ~]$ cinder list +--------------------------------------+-----------+------------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------+------+-------------+----------+-------------+ | 33737407-10a6-4e56-bcf5-666205d82c0c | available | encrypted volume | 1 | LUKS | false | | +--------------------------------------+-----------+------------------+------+-------------+----------+-------------+ [heat-admin@controller-0 ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2017-0319.html