Bug 1480363 - openstack-nova: unable to attach cinder volume to instance: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1
openstack-nova: unable to attach cinder volume to instance: libvirtError: int...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
unspecified Severity unspecified
: rc
: 12.0 (Pike)
Assigned To: Eoghan Glynn
Joe H. Rahme
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-10 15:53 EDT by Alexander Chuzhoy
Modified: 2017-08-22 05:31 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-22 05:31:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/containers/nova/nova-compute.log (137.04 KB, text/plain)
2017-08-10 16:53 EDT, Alexander Chuzhoy
no flags Details
/var/log/containers/nova/nova-compute.log with debug (22.05 KB, application/x-gzip)
2017-08-11 13:49 EDT, Alexander Chuzhoy
no flags Details

  None (edit)
Description Alexander Chuzhoy 2017-08-10 15:53:27 EDT
openstack-nova: unable to attach cinder volume to instance: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1

Environment:
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
openstack-nova-scheduler-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
python-nova-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-console-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
puppet-nova-11.3.0-0.20170805105252.30a205c.el7ost.noarch
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
openstack-nova-conductor-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-novncproxy-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
openstack-nova-compute-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-migration-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-api-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
openstack-nova-common-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64
python-novaclient-9.1.0-0.20170804194758.0a53d19.el7ost.noarch
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
openstack-nova-placement-api-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-daemon-config-network-3.2.0-14.el7.x86_64

openstack-tripleo-heat-templates-7.0.0-0.20170805163045.el7ost.noarch
instack-undercloud-7.2.1-0.20170729010705.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch


Steps to reproduce:
1. Deploy OC
2. Launch instance (tried with cirros and rhel image).
3. Create cinder volume
4. Try to attach the cinder volume to the instance

Result:
Although no error is shown in the console:
(overcloud) [stack@undercloud-0 ~]$ nova volume-attach rhel 2cf5dbe2-bbc3-42c8-b7ff-ea95c3dadc3d
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 2cf5dbe2-bbc3-42c8-b7ff-ea95c3dadc3d |
| serverId | 281edd5a-83de-4b4f-a301-329658ecf4be |
| volumeId | 2cf5dbe2-bbc3-42c8-b7ff-ea95c3dadc3d |


Checking the instance - there's no /dev/vdb.


Looking in nova-compute.log:
[root@overcloud-compute-0 ~]# grep -i error /var/log/containers/nova/nova-compute.log 
2017-08-10 19:31:22.214 1 ERROR nova.virt.libvirt.imagebackend [req-3530c749-84d1-4b13-8b74-80155fd5978f efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] /var/lib/nova/instances/_base/c62e48c870a2f6983224417418c841eca33de8ff virtual size 10737418240 larger than flavor root disk size 5368709120
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [req-3530c749-84d1-4b13-8b74-80155fd5978f efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: feae9666-9b8d-4d07-ab05-781ffca090e4] Instance failed to spawn: FlavorDiskSmallerThanImage: Flavor's disk is too small for requested image. Flavor disk is 5368709120 bytes, image is 10737418240 bytes.
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4] Traceback (most recent call last):
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     yield resources
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1940, in _build_and_run_instance
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     block_device_info=block_device_info)
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2793, in spawn
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     block_device_info=block_device_info)
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3231, in _create_image
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     fallback_from_host)
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3322, in _create_and_inject_local_root
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     instance, size, fallback_from_host)
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6968, in _try_fetch_image_cache
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     size=size)
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in cache
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     *args, **kwargs)
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 893, in create_image
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     self.verify_base_size(base, size)
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 297, in verify_base_size
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     flavor_size=size, image_size=base_size)
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4] FlavorDiskSmallerThanImage: Flavor's disk is too small for requested image. Flavor disk is 5368709120 bytes, image is 10737418240 bytes.
2017-08-10 19:31:22.216 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4] 
2017-08-10 19:31:24.441 1 ERROR nova.compute.manager [req-3530c749-84d1-4b13-8b74-80155fd5978f efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: feae9666-9b8d-4d07-ab05-781ffca090e4] Build of instance feae9666-9b8d-4d07-ab05-781ffca090e4 aborted: Flavor's disk is too small for requested image. Flavor disk is 5368709120 bytes, image is 10737418240 bytes.: BuildAbortException: Build of instance feae9666-9b8d-4d07-ab05-781ffca090e4 aborted: Flavor's disk is too small for requested image. Flavor disk is 5368709120 bytes, image is 10737418240 bytes.
2017-08-10 19:31:24.441 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4] Traceback (most recent call last):
2017-08-10 19:31:24.441 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1789, in _do_build_and_run_instance
2017-08-10 19:31:24.441 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     filter_properties)
2017-08-10 19:31:24.441 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2011, in _build_and_run_instance
2017-08-10 19:31:24.441 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4]     reason=e.format_message())
2017-08-10 19:31:24.441 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4] BuildAbortException: Build of instance feae9666-9b8d-4d07-ab05-781ffca090e4 aborted: Flavor's disk is too small for requested image. Flavor disk is 5368709120 bytes, image is 10737418240 bytes.
2017-08-10 19:31:24.441 1 ERROR nova.compute.manager [instance: feae9666-9b8d-4d07-ab05-781ffca090e4] 
2017-08-10 19:34:43.812 1 ERROR nova.virt.libvirt.imagebackend [req-a58a8e5f-4aa6-44a3-a529-360d8ece8939 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] /var/lib/nova/instances/_base/c62e48c870a2f6983224417418c841eca33de8ff virtual size 10737418240 larger than flavor root disk size 6442450944
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [req-a58a8e5f-4aa6-44a3-a529-360d8ece8939 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0] Instance failed to spawn: FlavorDiskSmallerThanImage: Flavor's disk is too small for requested image. Flavor disk is 6442450944 bytes, image is 10737418240 bytes.
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0] Traceback (most recent call last):
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     yield resources
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1940, in _build_and_run_instance
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     block_device_info=block_device_info)
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2793, in spawn
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     block_device_info=block_device_info)
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3231, in _create_image
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     fallback_from_host)
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3322, in _create_and_inject_local_root
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     instance, size, fallback_from_host)
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6968, in _try_fetch_image_cache
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     size=size)
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in cache
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     *args, **kwargs)
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 893, in create_image
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     self.verify_base_size(base, size)
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 297, in verify_base_size
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     flavor_size=size, image_size=base_size)
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0] FlavorDiskSmallerThanImage: Flavor's disk is too small for requested image. Flavor disk is 6442450944 bytes, image is 10737418240 bytes.
2017-08-10 19:34:43.814 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0] 
2017-08-10 19:34:45.034 1 ERROR nova.compute.manager [req-a58a8e5f-4aa6-44a3-a529-360d8ece8939 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0] Build of instance b8903e8f-68ae-4cf7-9a28-7f2245de60d0 aborted: Flavor's disk is too small for requested image. Flavor disk is 6442450944 bytes, image is 10737418240 bytes.: BuildAbortException: Build of instance b8903e8f-68ae-4cf7-9a28-7f2245de60d0 aborted: Flavor's disk is too small for requested image. Flavor disk is 6442450944 bytes, image is 10737418240 bytes.
2017-08-10 19:34:45.034 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0] Traceback (most recent call last):
2017-08-10 19:34:45.034 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1789, in _do_build_and_run_instance
2017-08-10 19:34:45.034 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     filter_properties)
2017-08-10 19:34:45.034 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2011, in _build_and_run_instance
2017-08-10 19:34:45.034 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0]     reason=e.format_message())
2017-08-10 19:34:45.034 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0] BuildAbortException: Build of instance b8903e8f-68ae-4cf7-9a28-7f2245de60d0 aborted: Flavor's disk is too small for requested image. Flavor disk is 6442450944 bytes, image is 10737418240 bytes.
2017-08-10 19:34:45.034 1 ERROR nova.compute.manager [instance: b8903e8f-68ae-4cf7-9a28-7f2245de60d0] 
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [req-a4455de3-c161-4ce0-8891-5aee8950d169 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Failed to attach volume at mountpoint: /dev/vdb: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Traceback (most recent call last):
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     guest.attach_device(conf, persistent=True, live=live)
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 308, in attach_device
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = execute(f, *args, **kwargs)
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(c, e, tb)
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = meth(*args, **kwargs)
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:13.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] 
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [req-a4455de3-c161-4ce0-8891-5aee8950d169 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Driver failed to attach volume 2cf5dbe2-bbc3-42c8-b7ff-ea95c3dadc3d at /dev/vdb: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Traceback (most recent call last):
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 389, in attach
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     device_type=self['device_type'], encryption=encryption)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1266, in attach_volume
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._disconnect_volume(connection_info, disk_dev, instance)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self.force_reraise()
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     guest.attach_device(conf, persistent=True, live=live)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 308, in attach_device
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = execute(f, *args, **kwargs)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(c, e, tb)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = meth(*args, **kwargs)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:13.308 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] 
2017-08-10 19:38:13.310 1 WARNING nova.volume.cinder [req-a4455de3-c161-4ce0-8891-5aee8950d169 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [req-a4455de3-c161-4ce0-8891-5aee8950d169 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Failed to attach 2cf5dbe2-bbc3-42c8-b7ff-ea95c3dadc3d at /dev/vdb: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Traceback (most recent call last):
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4767, in _attach_volume
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     do_driver_attach=True)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 45, in wrapped
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     ret_val = method(obj, context, *args, **kwargs)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 398, in attach
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     connector)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self.force_reraise()
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 389, in attach
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     device_type=self['device_type'], encryption=encryption)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1266, in attach_volume
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._disconnect_volume(connection_info, disk_dev, instance)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self.force_reraise()
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     guest.attach_device(conf, persistent=True, live=live)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 308, in attach_device
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = execute(f, *args, **kwargs)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(c, e, tb)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = meth(*args, **kwargs)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:13.839 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] 
2017-08-10 19:38:13.841 1 WARNING nova.volume.cinder [req-a4455de3-c161-4ce0-8891-5aee8950d169 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server [req-a4455de3-c161-4ce0-8891-5aee8950d169 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] Exception during message handling: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 211, in decorated_function
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 199, in decorated_function
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4752, in attach_volume
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4750, in do_attach_volume
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     bdm.destroy()
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4747, in do_attach_volume
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4781, in _attach_volume
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     volume_id=bdm.volume_id)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4767, in _attach_volume
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     do_driver_attach=True)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 45, in wrapped
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     ret_val = method(obj, context, *args, **kwargs)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 398, in attach
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     connector)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 389, in attach
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     device_type=self['device_type'], encryption=encryption)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1266, in attach_volume
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     self._disconnect_volume(connection_info, disk_dev, instance)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     guest.attach_device(conf, persistent=True, live=live)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 308, in attach_device
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:38:14.220 1 ERROR oslo_messaging.rpc.server 
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [req-8dd1e465-c2f6-49ba-b496-099536fc6149 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Failed to attach volume at mountpoint: /dev/vdb: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Traceback (most recent call last):
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     guest.attach_device(conf, persistent=True, live=live)
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 308, in attach_device
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = execute(f, *args, **kwargs)
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(c, e, tb)
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = meth(*args, **kwargs)
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.303 1 ERROR nova.virt.libvirt.driver [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] 
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [req-8dd1e465-c2f6-49ba-b496-099536fc6149 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Driver failed to attach volume 2cf5dbe2-bbc3-42c8-b7ff-ea95c3dadc3d at /dev/vdb: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Traceback (most recent call last):
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 389, in attach
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     device_type=self['device_type'], encryption=encryption)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1266, in attach_volume
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._disconnect_volume(connection_info, disk_dev, instance)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self.force_reraise()
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     guest.attach_device(conf, persistent=True, live=live)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 308, in attach_device
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = execute(f, *args, **kwargs)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(c, e, tb)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = meth(*args, **kwargs)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.304 1 ERROR nova.virt.block_device [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] 
2017-08-10 19:51:17.305 1 WARNING nova.volume.cinder [req-8dd1e465-c2f6-49ba-b496-099536fc6149 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [req-8dd1e465-c2f6-49ba-b496-099536fc6149 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Failed to attach 2cf5dbe2-bbc3-42c8-b7ff-ea95c3dadc3d at /dev/vdb: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] Traceback (most recent call last):
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4767, in _attach_volume
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     do_driver_attach=True)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 45, in wrapped
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     ret_val = method(obj, context, *args, **kwargs)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 398, in attach
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     connector)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self.force_reraise()
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 389, in attach
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     device_type=self['device_type'], encryption=encryption)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1266, in attach_volume
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._disconnect_volume(connection_info, disk_dev, instance)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self.force_reraise()
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     guest.attach_device(conf, persistent=True, live=live)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 308, in attach_device
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = execute(f, *args, **kwargs)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     six.reraise(c, e, tb)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     rv = meth(*args, **kwargs)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.766 1 ERROR nova.compute.manager [instance: 281edd5a-83de-4b4f-a301-329658ecf4be] 
2017-08-10 19:51:17.768 1 WARNING nova.volume.cinder [req-8dd1e465-c2f6-49ba-b496-099536fc6149 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server [req-8dd1e465-c2f6-49ba-b496-099536fc6149 efa8cb8e2d9e438784edc732713d54f5 74b09ed902344e229f620c1dde332906 - default default] Exception during message handling: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 211, in decorated_function
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 199, in decorated_function
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4752, in attach_volume
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4750, in do_attach_volume
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     bdm.destroy()
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4747, in do_attach_volume
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4781, in _attach_volume
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     volume_id=bdm.volume_id)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4767, in _attach_volume
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     do_driver_attach=True)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 45, in wrapped
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     ret_val = method(obj, context, *args, **kwargs)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 398, in attach
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     connector)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 389, in attach
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     device_type=self['device_type'], encryption=encryption)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1266, in attach_volume
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     self._disconnect_volume(connection_info, disk_dev, instance)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     guest.attach_device(conf, persistent=True, live=live)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 308, in attach_device
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2017-08-10 19:51:17.897 1 ERROR oslo_messaging.rpc.server
Comment 1 Alexander Chuzhoy 2017-08-10 16:53 EDT
Created attachment 1311907 [details]
/var/log/containers/nova/nova-compute.log
Comment 2 Gorka Eguileor 2017-08-11 04:30:23 EDT
This seems to be trying to store a 10GiB image into a 5GiB disk:

  FlavorDiskSmallerThanImage: Flavor's disk is too small for requested image. Flavor disk is 5368709120 bytes, image is 10737418240 bytes.
Comment 3 Alexander Chuzhoy 2017-08-11 09:06:24 EDT
(In reply to Gorka Eguileor from comment #2)
> This seems to be trying to store a 10GiB image into a 5GiB disk:
> 
>   FlavorDiskSmallerThanImage: Flavor's disk is too small for requested
> image. Flavor disk is 5368709120 bytes, image is 10737418240 bytes.

We saw that error initially, but then updated the flavor to 12G and still failed.
Beforehand, we tried with cirros image (very small and used to work in previous releases)- also failed.
Comment 4 John Fulton 2017-08-11 11:22:52 EDT
Just want to clarify: this is not about fitting images into disks at boot. That happened earlier so it will be in the logs but we got past that. What this bug is about is:

Attaching an existing volume to an already running instance
Comment 5 Alexander Chuzhoy 2017-08-11 13:49 EDT
Created attachment 1312211 [details]
/var/log/containers/nova/nova-compute.log with debug
Comment 7 Alexander Chuzhoy 2017-08-18 11:20:34 EDT
Was able to attach the volume on deployment with internal ceph.

Environment:
instack-undercloud-7.2.1-0.20170729010706.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
ceph-ansible-3.0.0-0.1.rc3.el7cp.noarch

ceph-mon-10.2.7-28.el7cp.x86_64
ceph-osd-10.2.7-28.el7cp.x86_64
libcephfs1-10.2.7-28.el7cp.x86_64
puppet-ceph-2.3.1-0.20170805094345.868e6d6.el7ost.noarch
ceph-selinux-10.2.7-28.el7cp.x86_64
python-cephfs-10.2.7-28.el7cp.x86_64
ceph-base-10.2.7-28.el7cp.x86_64
ceph-radosgw-10.2.7-28.el7cp.x86_64
ceph-mds-10.2.7-28.el7cp.x86_64
ceph-common-10.2.7-28.el7cp.x86_64



2017-08-18 15:12:14.752 1 INFO nova.compute.manager [req-97485f97-c26e-4cfa-b62c-fe98068c9887 4ea007d1ef484f7c935667ccaea99d74 502bd4c746314367b1b31dde42d8034b - default default] [instance: 1fd7fcdc-4bb6-415f-878e-1b8d1e644d85] Attaching volume f9df1e73-e6e5-4015-b244-d3b0906cb61e to /dev/vdb
2017-08-18 15:12:14.821 1 WARNING nova.volume.cinder [req-97485f97-c26e-4cfa-b62c-fe98068c9887 4ea007d1ef484f7c935667ccaea99d74 502bd4c746314367b1b31dde42d8034b - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.
2017-08-18 15:12:15.873 1 INFO oslo.privsep.daemon [req-97485f97-c26e-4cfa-b62c-fe98068c9887 4ea007d1ef484f7c935667ccaea99d74 502bd4c746314367b1b31dde42d8034b - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--config-file', '/etc/nova/rootwrap.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpNRLmpo/privsep.sock']
2017-08-18 15:12:16.466 1 INFO oslo.privsep.daemon [req-97485f97-c26e-4cfa-b62c-fe98068c9887 4ea007d1ef484f7c935667ccaea99d74 502bd4c746314367b1b31dde42d8034b - default default] Spawned new privsep daemon via rootwrap
2017-08-18 15:12:16.407 1056 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-08-18 15:12:16.411 1056 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-08-18 15:12:16.413 1056 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
2017-08-18 15:12:16.413 1056 INFO oslo.privsep.daemon [-] privsep daemon running as pid 1056
2017-08-18 15:12:16.646 1 WARNING nova.volume.cinder [req-97485f97-c26e-4cfa-b62c-fe98068c9887 4ea007d1ef484f7c935667ccaea99d74 502bd4c746314367b1b31dde42d8034b - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.
2017-08-18 15:12:18.346 1 WARNING nova.volume.cinder [req-97485f97-c26e-4cfa-b62c-fe98068c9887 4ea007d1ef484f7c935667ccaea99d74 502bd4c746314367b1b31dde42d8034b - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.
2017-08-18 15:13:05.747 1 INFO nova.compute.resource_tracker [req-95fb4fe9-fd48-41be-abdb-29fbf98d29ad - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=6143MB used_ram=4608MB phys_disk=118GB used_disk=1GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-08-18 15:14:05.738 1 INFO nova.compute.resource_tracker [req-95fb4fe9-fd48-41be-abdb-29fbf98d29ad - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=6143MB used_ram=4608MB phys_disk=118GB used_disk=1GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-08-18 15:15:07.769 1 INFO nova.compute.resource_tracker [req-95fb4fe9-fd48-41be-abdb-29fbf98d29ad - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=6143MB used_ram=4608MB phys_disk=118GB used_disk=1GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-08-18 15:16:09.733 1 INFO nova.compute.resource_tracker [req-95fb4fe9-fd48-41be-abdb-29fbf98d29ad - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=6143MB used_ram=4608MB phys_disk=118GB used_disk=1GB total_vcpus=4 used_vcpus=1 pci_stats=[]




[stack@undercloud-0 ~]$ ssh cirros@10.0.0.194 "sudo /sbin/fdisk -l"
cirros@10.0.0.194's password: 

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *       16065     2088449     1036192+  83  Linux

Disk /dev/vdb: 1073 MB, 1073741824 bytes
16 heads, 63 sectors/track, 2080 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/vdb doesn't contain a valid partition table
Comment 8 Giulio Fidente 2017-08-18 11:45:48 EDT
I believe this could be due to a mismatch in the volume features supported by the OSP containers (previously using Ceph libs from baseos, 0.94) with the version of Ceph running in the ceph containers (10.2).

Could you try, after creating a cinder volume, to run the following from one of the controller nodes:

# rbd ls volumes
# rbd info -p volumes $name_of_an_image_from_previous_command

The output should tell us which features are set for the image. Ideally we should repeat the same with older OSP container images and see if the enabled features are different.
Comment 10 Alexander Chuzhoy 2017-08-21 12:55:51 EDT
Works for me now.
Comment 11 Lee Yarwood 2017-08-22 05:31:58 EDT
(In reply to Alexander Chuzhoy from comment #10)
> Works for me now.

Closing this out as NOTABUG given this feedback, feel free to provide more detail and/or reopen this bug if we are able to reproduce this.

Note You need to log in before you can comment on or make changes to this bug.