1510339 – Fail to attach volume to the instance from different Ceph cluster backend

Bug 1510339 - Fail to attach volume to the instance from different Ceph cluster backend

Summary: Fail to attach volume to the instance from different Ceph cluster backend

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	10.0 (Newton)
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	z8
Target Release:	10.0 (Newton)
Assignee:	Lee Yarwood
QA Contact:	Joe H. Rahme
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-11-07 09:18 UTC by Shang Wu
Modified:	2019-09-09 16:06 UTC (History)
CC List:	11 users (show)
Fixed In Version:	openstack-nova-14.1.0-9.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-05-17 15:33:13 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
cinder.conf file (158.12 KB, text/plain) 2017-11-07 09:24 UTC, Shang Wu	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
OpenStack gerrit	389399	0	None	None	None	2017-11-07 09:18:55 UTC
Red Hat Product Errata	RHBA-2018:1595	0	None	None	None	2018-05-17 15:34:56 UTC

Description Shang Wu 2017-11-07 09:18:56 UTC

Description of problem:

Using Cinder to connect two different Ceph cluster. Each type named: Ceph and Cephnew.
Nova type name Ceph for ceph cluster.


Version-Release number of selected component (if applicable):
OSP 10

How reproducible:
100%

Steps to Reproduce:

1. Create volume on both ceph cluster:
1a. ceph1 for ceph cluster
1b. cephnew for cephnew cluster

2. Create an instance using volume

3. Attach volume from ceph1 to instance => Success

4. Attach volume cephnew to instance => Failed. (Error message from nova-compute log attached below. No obvious error from cinder log)
Actual results:

nova compute error log:
2017-10-13 11:37:30.407 29961 INFO nova.compute.manager [req-5ba8c9fc-6bb7-42fe-
a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e

b0201f0597f5432c98f63409ea685925 - - -] [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Attaching volume 9854875f-f062-4901-a627-
c69392c5f51e to /dev/vdb
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Failed to attach volume at mountpoint: /
dev/vdb
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Traceback (most recent call last):
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1173, in attach_volume
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] guest.attach_device(conf,
persistent=True, live=live)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/guest.py", line 294, in attach_device
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 186, in doit
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] result = proxy_call(self._autowrap, f,
*args, **kwargs)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 144, in proxy_call
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = execute(f, *args, **kwargs)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 125, in execute
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(c, e, tb)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 83, in tworker
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = meth(*args, **kwargs)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib64/python2.7/site-
packages/libvirt.py", line 560, in attachDeviceFlags

2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] if ret == -1: raise libvirtError
('virDomainAttachDeviceFlags() failed', dom=self)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] libvirtError: internal error: unable to
execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could
not be initialized
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] Driver failed to attach volume 9854875f-
f062-4901-a627-c69392c5f51e at /dev/vdb

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] Driver failed to attach volume 9854875f-
f062-4901-a627-c69392c5f51e at /dev/vdb

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Traceback (most recent call last):
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/block_device.py", line 274, in attach
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] device_type=self['device_type'],
encryption=encryption)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1184, in attach_volume
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._disconnect_volume(connection_info, disk_dev)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] self.force_reraise()
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(self.type_, self.value,
self.tb)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1173, in attach_volume
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] guest.attach_device(conf,
persistent=True, live=live)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/guest.py", line 294, in attach_device
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 186, in doit
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] result = proxy_call(self._autowrap, f,
*args, **kwargs)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 144, in proxy_call
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = execute(f, *args, **kwargs)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 125, in execute
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(c, e, tb)

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 83, in tworker
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = meth(*args, **kwargs)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib64/python2.7/site-
packages/libvirt.py", line 560, in attachDeviceFlags

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = meth(*args, **kwargs)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib64/python2.7/site-
packages/libvirt.py", line 560, in attachDeviceFlags

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] if ret == -1: raise libvirtError
('virDomainAttachDeviceFlags() failed', dom=self)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] libvirtError: internal error: unable to
execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could
not be initialized
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Failed to attach 9854875f-f062-4901-a627-
c69392c5f51e at /dev/vdb
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Traceback (most recent call last):
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/compute/manager.py", line 4685, in _attach_volume
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] do_check_attach=False,
do_driver_attach=True)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/block_device.py", line 49, in wrapped
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] ret_val = method(obj, context, *args,
**kwargs)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/block_device.py", line 283, in attach
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] connector)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] self.force_reraise()
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(self.type_, self.value,
self.tb)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/block_device.py", line 274, in attach

2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] device_type=self['device_type'],
encryption=encryption)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1184, in attach_volume
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._disconnect_volume(connection_info, disk_dev)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] self.force_reraise()
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(self.type_, self.value,
self.tb)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(self.type_, self.value,
self.tb)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1173, in attach_volume
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] guest.attach_device(conf,
persistent=True, live=live)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/guest.py", line 294, in attach_device
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 186, in doit
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] result = proxy_call(self._autowrap, f,
*args, **kwargs)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 144, in proxy_call
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = execute(f, *args, **kwargs)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 125, in execute
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(c, e, tb)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 83, in tworker
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = meth(*args, **kwargs)

2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib64/python2.7/site-
packages/libvirt.py", line 560, in attachDeviceFlags

2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] if ret == -1: raise libvirtError
('virDomainAttachDeviceFlags() failed', dom=self)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] libvirtError: internal error: unable to
execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could
not be initialized
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] Exception during message handling
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in
_process_incoming
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in
_do_dispatch
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in
_do_dispatch
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server function_name,
call_dict, binary)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 216, in
decorated_function
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 204, in
decorated_function
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4675, in attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
do_attach_volume(context, instance, driver_bdm)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4673, in do_attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server bdm.destroy()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4670, in do_attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4670, in do_attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return
self._attach_volume(context, instance, driver_bdm)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4693, in _attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.volume_api.unreserve_volume(context, bdm.volume_id)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4685, in _attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
do_check_attach=False, do_driver_attach=True)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/block_device.py", line 49, in wrapped
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server ret_val =
method(obj, context, *args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/block_device.py", line 283, in attach

2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server connector)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/block_device.py", line 274, in attach
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
device_type=self['device_type'], encryption=encryption)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/libvirt/driver.py", line 1184, in
attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self._disconnect_volume(connection_info, disk_dev)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/libvirt/driver.py", line 1173, in
attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
guest.attach_device(conf, persistent=True, live=live)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/libvirt/driver.py", line 1173, in
attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
guest.attach_device(conf, persistent=True, live=live)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/libvirt/guest.py", line 294, in attach_device
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server result =
proxy_call(self._autowrap, f, *args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server rv =
execute(f, *args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server six.reraise(c,
e, tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server rv =
meth(*args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/
lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server if ret == -1:
raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)

2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server libvirtError:
internal error: unable to execute QEMU command '__com.redhat_drive_add': Device
'drive-virtio-disk1' could not be initialized
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server

Expected results:
Volume successfully attached.

Additional info:
Found the possible fix from the upstream. Not sure if it is applied in later release already nor backported to OSP 10 or not.

Comment 1 Shang Wu 2017-11-07 09:24:07 UTC

Created attachment 1348875 [details]
cinder.conf file

Attach cinder.conf

Comment 2 Lee Yarwood 2017-11-10 10:09:21 UTC

The attached upstream change looks correct, I've posted these downstream and will test before asking for review.

Comment 11 errata-xmlrpc 2018-05-17 15:33:13 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1595

Note You need to log in before you can comment on or make changes to this bug.