Bug 1388417
| Summary: | Failed to detach an encrypted volume | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Attila Fazekas <afazekas> | ||||
| Component: | openstack-nova | Assignee: | Lee Yarwood <lyarwood> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Prasanth Anbalagan <panbalag> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 10.0 (Newton) | CC: | berrange, dasmith, ddomingo, dsariel, eglynn, eharney, jschluet, kchamart, lyarwood, sbauza, sferdjao, sgordon, srevivo, vromanso | ||||
| Target Milestone: | z2 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 10.0 (Newton) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-nova-14.0.3-4.el7ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-02-23 21:14:33 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
This landed in master before the break so I'm reusing this bug to track it into stable/newton and OSP 10. upstream/stable/newton patch just landed 2017-01-25 Verified as follows,
********
VERSION
********
[heat-admin@controller-0 ~]$ yum list installed | grep openstack-nova
openstack-nova-api.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed
openstack-nova-cert.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed
openstack-nova-common.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed
openstack-nova-compute.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed
openstack-nova-conductor.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed
openstack-nova-console.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed
openstack-nova-novncproxy.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed
openstack-nova-scheduler.noarch 1:14.0.3-8.el7ost @rhos-10.0-signed
*******
LOGS
*******
[heat-admin@controller-0 ~]$ cinder type-create LUKS
+--------------------------------------+------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+------+-------------+-----------+
| 5b0b0556-47fe-46f3-b645-4631840dc49c | LUKS | - | True |
+--------------------------------------+------+-------------+-----------+
[heat-admin@controller-0 ~]$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 \
> --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
| 5b0b0556-47fe-46f3-b645-4631840dc49c | nova.volume.encryptors.luks.LuksEncryptor | aes-xts-plain64 | 512 | front-end |
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
[heat-admin@controller-0 ~]$ cinder create --display-name 'encrypted volume' --volume-type LUKS 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-02-15T20:16:35.000000 |
| description | None |
| encrypted | True |
| id | 33737407-10a6-4e56-bcf5-666205d82c0c |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | encrypted volume |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 2fbbb659cb554fb3adffbdb2a127499f |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | 200007ec0598452c8d02fcf829a42850 |
| volume_type | LUKS |
+--------------------------------+--------------------------------------+
[heat-admin@controller-0 ~]$ cinder list
+--------------------------------------+-----------+------------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------------------+------+-------------+----------+-------------+
| 33737407-10a6-4e56-bcf5-666205d82c0c | available | encrypted volume | 1 | LUKS | false | |
+--------------------------------------+-----------+------------------+------+-------------+----------+-------------+
[heat-admin@controller-0 ~]$ nova list
+--------------------------------------+------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+-------------------+
| f27ce3b7-ec48-4b89-aeda-7980325347de | vm | ACTIVE | - | Running | public=10.0.0.215 |
+--------------------------------------+------+--------+------------+-------------+-------------------+
[heat-admin@controller-0 ~]$
[heat-admin@controller-0 ~]$ sudo grep "fixed_key" /etc/nova/nova.conf
fixed_key=8b9aacd510dcb09fdaacf684b22e9eec9d199c45e8ff1e75e8541c733f5fbbe3
[heat-admin@controller-0 ~]$
[heat-admin@controller-0 ~]$ sudo service openstack-nova-api restart
Redirecting to /bin/systemctl restart openstack-nova-api.service
[heat-admin@controller-0 ~]$ sudo service openstack-nova-cert restart
Redirecting to /bin/systemctl restart openstack-nova-cert.service
[heat-admin@controller-0 ~]$ sudo service openstack-nova-consoleauth restart
Redirecting to /bin/systemctl restart openstack-nova-consoleauth.service
[heat-admin@controller-0 ~]$ sudo service openstack-nova-scheduler restart
Redirecting to /bin/systemctl restart openstack-nova-scheduler.service
[heat-admin@controller-0 ~]$ sudo service openstack-nova-conductor restart
Redirecting to /bin/systemctl restart openstack-nova-conductor.service
[heat-admin@controller-0 ~]$ sudo service openstack-nova-novncproxy restart
Redirecting to /bin/systemctl restart openstack-nova-novncproxy.service
[heat-admin@controller-0 ~]$
[heat-admin@compute-0 ~]$ sudo grep fixed_key /etc/nova/nova.conf
fixed_key=8b9aacd510dcb09fdaacf684b22e9eec9d199c45e8ff1e75e8541c733f5fbbe3
[heat-admin@compute-0 ~]$ sudo service openstack-nova-compute restart
Redirecting to /bin/systemctl restart openstack-nova-compute.service
[heat-admin@compute-0 ~]$
[heat-admin@controller-0 ~]$
[heat-admin@controller-0 ~]$ nova volume-attach vm 33737407-10a6-4e56-bcf5-666205d82c0c
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | 33737407-10a6-4e56-bcf5-666205d82c0c |
| serverId | f27ce3b7-ec48-4b89-aeda-7980325347de |
| volumeId | 33737407-10a6-4e56-bcf5-666205d82c0c |
+----------+--------------------------------------+
[heat-admin@controller-0 ~]$ cinder list
+--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+
| 33737407-10a6-4e56-bcf5-666205d82c0c | in-use | encrypted volume | 1 | LUKS | false | f27ce3b7-ec48-4b89-aeda-7980325347de |
+--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+
[heat-admin@controller-0 ~]$
[heat-admin@controller-0 ~]$ nova volume-detach vm 33737407-10a6-4e56-bcf5-666205d82c0c
[heat-admin@controller-0 ~]$
[heat-admin@controller-0 ~]$ cinder list
+--------------------------------------+-----------+------------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------------------+------+-------------+----------+-------------+
| 33737407-10a6-4e56-bcf5-666205d82c0c | available | encrypted volume | 1 | LUKS | false | |
+--------------------------------------+-----------+------------------+------+-------------+----------+-------------+
[heat-admin@controller-0 ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2017-0319.html |
Created attachment 1213817 [details] nova-compute.log Description of problem: packstack / nova / simple keymgr (lvm) setup fails to detach the encrypted volume. Fails on the tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup[compute,id-cbc752ed-b716-4717-910f-956cce965722,image,volume] test. Version-Release number of selected component (if applicable): puppet-nova-9.4.0-1.el7ost.noarch openstack-packstack-puppet-9.0.0-0.4.0rc4.el7ost.noarch python-crypto-2.6.1-1.1.el7.x86_64 libgcrypt-devel-1.5.3-12.el7_1.1.x86_64 openstack-nova-scheduler-14.0.1-3.el7ost.noarch python-novaclient-6.0.0-1.el7ost.noarch openstack-nova-compute-14.0.1-3.el7ost.noarch openstack-nova-common-14.0.1-3.el7ost.noarch openstack-nova-novncproxy-14.0.1-3.el7ost.noarch openstack-nova-cert-14.0.1-3.el7ost.noarch libgcrypt-1.5.3-12.el7_1.1.x86_64 python2-cryptography-1.3.1-3.el7.x86_64 m2crypto-0.21.1-17.el7.x86_64 openstack-packstack-9.0.0-0.4.0rc4.el7ost.noarch lvm2-libs-2.02.166-1.el7.x86_64 openstack-nova-conductor-14.0.1-3.el7ost.noarch openstack-nova-console-14.0.1-3.el7ost.noarch openstack-nova-api-14.0.1-3.el7ost.noarch cryptsetup-libs-1.7.2-1.el7.x86_64 cryptsetup-1.7.2-1.el7.x86_64 lvm2-2.02.166-1.el7.x86_64 python-nova-14.0.1-3.el7ost.noarch How reproducible: frequently Steps to Reproduce: 1. create packstack setup (nothing special) 2. add 64 char key (hexdump -n 32 -v -e '/1 "%02x"' /dev/urandom) to the /etc/nova/nova.conf [key_manager] fixed_key 3. restart the nova compute 4. run tempest test_encrypted_cinder_volumes test (for ex.: ostestr -r test_encrypted_cinder_volumes) Actual results: test failed: failed to reach available status (current in-use) within the required time (300 s). Expected results: test_encrypted_cinder_volumes passes Additional info: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup remove crypt-ip-192.168.1.13:3260-iscsi-iqn.2010-10.org.openstack:volume-9439e922-1051-4d83-87c7-172689ac29da-lun-0 failed according to the nova-compute.log . remove ioctl on crypt-ip-192.168.1.13:3260-iscsi-iqn.2010-10.org.openstack:volume-9439e922-1051-4d83-87c7-172689ac29da-lun-0 failed: Device or resource busy The issue can be lvm/libvirt (related service unit) configuration issue as well, but it is also possible the nova has to wait a little before it can safely use `cryptsetup remove`. libvirt likely asked for removing the disk before `cryptsetup remove` part, it just not completed.