Bug 1510339

Summary:

Fail to attach volume to the instance from different Ceph cluster backend

Product:

Red Hat OpenStack

Reporter:

Shang Wu <shwu>

Component:

openstack-nova

Assignee:

Lee Yarwood <lyarwood>

Status:

CLOSED ERRATA

QA Contact:

Joe H. Rahme <jhakimra>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

10.0 (Newton)

CC:

awaugama, berrange, dasmith, eglynn, kchamart, lyarwood, sbauza, sferdjao, sgordon, srevivo, vromanso

Target Milestone:

Keywords:

Triaged, ZStream

Target Release:

10.0 (Newton)

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

openstack-nova-14.1.0-9.el7ost

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2018-05-17 15:33:13 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
cinder.conf file	none

Description Shang Wu 2017-11-07 09:18:56 UTC

Description of problem:

Using Cinder to connect two different Ceph cluster. Each type named: Ceph and Cephnew.
Nova type name Ceph for ceph cluster.


Version-Release number of selected component (if applicable):
OSP 10

How reproducible:
100%

Steps to Reproduce:

1. Create volume on both ceph cluster:
1a. ceph1 for ceph cluster
1b. cephnew for cephnew cluster

2. Create an instance using volume

3. Attach volume from ceph1 to instance => Success

4. Attach volume cephnew to instance => Failed. (Error message from nova-compute log attached below. No obvious error from cinder log)
Actual results:

nova compute error log:
2017-10-13 11:37:30.407 29961 INFO nova.compute.manager [req-5ba8c9fc-6bb7-42fe-
a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e

b0201f0597f5432c98f63409ea685925 - - -] [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Attaching volume 9854875f-f062-4901-a627-
c69392c5f51e to /dev/vdb
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Failed to attach volume at mountpoint: /
dev/vdb
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Traceback (most recent call last):
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1173, in attach_volume
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] guest.attach_device(conf,
persistent=True, live=live)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/guest.py", line 294, in attach_device
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 186, in doit
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] result = proxy_call(self._autowrap, f,
*args, **kwargs)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 144, in proxy_call
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = execute(f, *args, **kwargs)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 125, in execute
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(c, e, tb)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 83, in tworker
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = meth(*args, **kwargs)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib64/python2.7/site-
packages/libvirt.py", line 560, in attachDeviceFlags

2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] if ret == -1: raise libvirtError
('virDomainAttachDeviceFlags() failed', dom=self)
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] libvirtError: internal error: unable to
execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could
not be initialized
2017-10-13 11:37:30.792 29961 ERROR nova.virt.libvirt.driver [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] Driver failed to attach volume 9854875f-
f062-4901-a627-c69392c5f51e at /dev/vdb

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] Driver failed to attach volume 9854875f-
f062-4901-a627-c69392c5f51e at /dev/vdb

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Traceback (most recent call last):
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/block_device.py", line 274, in attach
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] device_type=self['device_type'],
encryption=encryption)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1184, in attach_volume
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._disconnect_volume(connection_info, disk_dev)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] self.force_reraise()
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(self.type_, self.value,
self.tb)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1173, in attach_volume
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] guest.attach_device(conf,
persistent=True, live=live)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/guest.py", line 294, in attach_device
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 186, in doit
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] result = proxy_call(self._autowrap, f,
*args, **kwargs)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 144, in proxy_call
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = execute(f, *args, **kwargs)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 125, in execute
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(c, e, tb)

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 83, in tworker
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = meth(*args, **kwargs)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib64/python2.7/site-
packages/libvirt.py", line 560, in attachDeviceFlags

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = meth(*args, **kwargs)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib64/python2.7/site-
packages/libvirt.py", line 560, in attachDeviceFlags

2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] if ret == -1: raise libvirtError
('virDomainAttachDeviceFlags() failed', dom=self)
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] libvirtError: internal error: unable to
execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could
not be initialized
2017-10-13 11:37:30.793 29961 ERROR nova.virt.block_device [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Failed to attach 9854875f-f062-4901-a627-
c69392c5f51e at /dev/vdb
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] Traceback (most recent call last):
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/compute/manager.py", line 4685, in _attach_volume
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] do_check_attach=False,
do_driver_attach=True)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/block_device.py", line 49, in wrapped
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] ret_val = method(obj, context, *args,
**kwargs)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/block_device.py", line 283, in attach
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] connector)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] self.force_reraise()
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(self.type_, self.value,
self.tb)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/block_device.py", line 274, in attach

2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] device_type=self['device_type'],
encryption=encryption)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1184, in attach_volume
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._disconnect_volume(connection_info, disk_dev)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] self.force_reraise()
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(self.type_, self.value,
self.tb)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(self.type_, self.value,
self.tb)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/driver.py", line 1173, in attach_volume
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] guest.attach_device(conf,
persistent=True, live=live)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
nova/virt/libvirt/guest.py", line 294, in attach_device
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 186, in doit
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] result = proxy_call(self._autowrap, f,
*args, **kwargs)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 144, in proxy_call
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = execute(f, *args, **kwargs)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 125, in execute
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] six.reraise(c, e, tb)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib/python2.7/site-packages/
eventlet/tpool.py", line 83, in tworker
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] rv = meth(*args, **kwargs)

2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:

07da78ed-2dd4-4758-9ed3-7837399346cf] File "/usr/lib64/python2.7/site-
packages/libvirt.py", line 560, in attachDeviceFlags

2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] if ret == -1: raise libvirtError
('virDomainAttachDeviceFlags() failed', dom=self)
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf] libvirtError: internal error: unable to
execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could
not be initialized
2017-10-13 11:37:30.937 29961 ERROR nova.compute.manager [instance:
07da78ed-2dd4-4758-9ed3-7837399346cf]
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
[req-5ba8c9fc-6bb7-42fe-a3b7-5610059b4b5e 1fba8452348c4537927a9c44b9fffc2e
b0201f0597f5432c98f63409ea685925 - - -] Exception during message handling
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in
_process_incoming
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in
_do_dispatch
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in
_do_dispatch
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server function_name,
call_dict, binary)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 216, in
decorated_function
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 204, in
decorated_function
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4675, in attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
do_attach_volume(context, instance, driver_bdm)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4673, in do_attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server bdm.destroy()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4670, in do_attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4670, in do_attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server return
self._attach_volume(context, instance, driver_bdm)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4693, in _attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.volume_api.unreserve_volume(context, bdm.volume_id)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/compute/manager.py", line 4685, in _attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
do_check_attach=False, do_driver_attach=True)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/block_device.py", line 49, in wrapped
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server ret_val =
method(obj, context, *args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/block_device.py", line 283, in attach

2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server connector)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/block_device.py", line 274, in attach
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
device_type=self['device_type'], encryption=encryption)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/libvirt/driver.py", line 1184, in
attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self._disconnect_volume(connection_info, disk_dev)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/libvirt/driver.py", line 1173, in
attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
guest.attach_device(conf, persistent=True, live=live)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/libvirt/driver.py", line 1173, in
attach_volume
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
guest.attach_device(conf, persistent=True, live=live)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/nova/virt/libvirt/guest.py", line 294, in attach_device
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server
self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server result =
proxy_call(self._autowrap, f, *args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server rv =
execute(f, *args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server six.reraise(c,
e, tb)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/lib/
python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server rv =
meth(*args, **kwargs)
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server File "/usr/
lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server if ret == -1:
raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)

2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server libvirtError:
internal error: unable to execute QEMU command '__com.redhat_drive_add': Device
'drive-virtio-disk1' could not be initialized
2017-10-13 11:37:31.001 29961 ERROR oslo_messaging.rpc.server

Expected results:
Volume successfully attached.

Additional info:
Found the possible fix from the upstream. Not sure if it is applied in later release already nor backported to OSP 10 or not.

Comment 1 Shang Wu 2017-11-07 09:24:07 UTC

Created attachment 1348875 [details]
cinder.conf file

Attach cinder.conf

Comment 2 Lee Yarwood 2017-11-10 10:09:21 UTC

The attached upstream change looks correct, I've posted these downstream and will test before asking for review.

Comment 11 errata-xmlrpc 2018-05-17 15:33:13 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1595