Previously, rebuilding an instance with a volume attached would leave the volume in an inconsistent state; the instance would be rebuilt without the volume attached, and the volume could not be attached to any instance. This was because during rebuilds, the Compute service did not check for any changes in block device mappings to attached volumes first.
With this fix, the Compute service now checks for new block device mappings during rebuilds. When rebuilding an instance, any attached volumes will be reattached after the rebuild process.
Created attachment 813796[details]
logs
Description of problem:
I had an instance with no volume attached and I took a backup of the instance
after taking the backup I attached a volume and rebuild the instance from the backup.
once the rebuild has finished, the volume no longer appears as attached.
when we try to attach it to the same device it will give an error that the device is already in use and if we try to attach it to a different device it appears as if it works but when we try to detach the device we get DiskNotFound: No disk at vdc (which is the original device).
Version-Release number of selected component (if applicable):
openstack-nova-compute-2013.2-0.25.rc1.el6ost.noarch
How reproducible:
100%
Steps to Reproduce:
1. launch an instance from an image and take a backup of the instance
2. attach a volume to the instance
3. rebuild the instance from the backup without detaching the volume first
4. try to re-attach the volume to the instance on the same device
5. attach the volume to the instance on a different device
6. detach the volume
Actual results:
1. after the rebuild the volume no longer appears as attached
2. when we try to attach to the same device, although the volume does not appear as attached, it is actually still attached in the instance
3. we can attach the volume to a different device on the same instance
4. when we try to detach it, the detach is sent to the original device and we fail to detach with No Disk error.
Expected results:
if we allow rebuild of instance with a volume we need to either detach completely or not detach at all
Additional info: logs
2013-10-18 16:54:36.632 2478 DEBUG qpid.messaging [-] RETR[2fca830]: Message(properties={'x-amqp-0-10.routing-key': u'reply_70fb16e321724b38b3d3face4e83f363'}, content={u'oslo.message': u'{"_unique_id": "dd2a85b63c56498c8f2835f9b96e9bb9"
, "failure": null, "_msg_id": "7ce524cebbb34aecab9d608a48103a1c", "result": null, "ending": true}', u'oslo.version': u'2.0'}) _get /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py:654
2013-10-18 16:54:36.633 2478 DEBUG qpid.messaging.io.ops [-] SENT[2fa6cb0]: MessageFlow(destination='0', unit=0, value=1L, id=serial(5206)) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686
2013-10-18 16:54:36.634 2478 DEBUG qpid.messaging.io.ops [-] SENT[2fa6cb0]: SessionCompleted(commands=[0-5199]) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686
2013-10-18 16:54:36.639 2478 ERROR nova.openstack.common.rpc.amqp [req-4a884bb4-5ba6-403d-8be7-df1eeebc1324 bbce236d5aac4d1dbc086a8835ed0ebc d09f3bf0f9224affa92ab97010b37270] Exception during message handling
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp **args)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp pass
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3624, in reserve_block_device_name
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return do_reserve()
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3613, in do_reserve
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp context, instance, bdms, device)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/utils.py", line 135, in get_device_name_for_instance
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp mappings['root'], device)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/utils.py", line 217, in get_next_device_name
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp raise exception.DevicePathInUse(path=device)
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp DevicePathInUse: The supplied device path (/dev/vdc) is in use.
2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp
2013-10-18 16:54:36.641 2478 ERROR nova.openstack.common.rpc.common [req-4a884bb4-5ba6-403d-8be7-df1eeebc1324 bbce236d5aac4d1dbc086a8835ed0ebc d09f3bf0f9224affa92ab97010b37270] Returning exception The supplied device path (/dev/vdc) is i
n use. to caller
2013-10-18 16:54:36.642 2478 ERROR nova.openstack.common.rpc.common [req-4a884bb4-5ba6-403d-8be7-df1eeebc1324 bbce236d5aac4d1dbc086a8835ed0ebc d09f3bf0f9224affa92ab97010b37270] ['Traceback (most recent call last):\n', ' File "/usr/lib/p
ython2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data\n **args)\n', ' File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyo
bj, method)(ctxt, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped\n payload)\n', ' File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped\n return f(self, con
text, *args, **kw)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function\n pass\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function\
n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib/python2.6/site-packages/nova/compu
te/manager.py", line 258, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3624, in reserve_block_device_name\n return do_reserve()
:
2013-10-18 16:58:11.289 2478 DEBUG qpid.messaging [-] RETR[2fca830]: Message(properties={'x-amqp-0-10.routing-key': u'reply_70fb16e321724b38b3d3face4e83f363'}, content={u'oslo.message': u'{"_unique_id": "20a035f66eac46f7b44e8420cd5a2d14"
, "failure": null, "_msg_id": "27d7abfd907146c4a0e052cf067a4d69", "result": null, "ending": true}', u'oslo.version': u'2.0'}) _get /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py:654
2013-10-18 16:58:11.289 2478 DEBUG qpid.messaging.io.ops [-] SENT[2fa6cb0]: MessageFlow(destination='0', unit=0, value=1L, id=serial(5308)) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686
2013-10-18 16:58:11.290 2478 DEBUG qpid.messaging.io.ops [-] SENT[2fa6cb0]: SessionCompleted(commands=[0-5301]) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686
2013-10-18 16:58:11.296 2478 ERROR nova.openstack.common.rpc.amqp [req-62e56d8a-9dc1-447c-acef-2b2dafdd5079 bbce236d5aac4d1dbc086a8835ed0ebc d09f3bf0f9224affa92ab97010b37270] Exception during message handling
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp **args)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp pass
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3760, in detach_volume
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp self._detach_volume(context, instance, bdm)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3732, in _detach_volume
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp self.volume_api.roll_detaching(context, volume_id)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3725, in _detach_volume
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp encryption=encryption)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1205, in detach_volume
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp raise exception.DiskNotFound(location=disk_dev)
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp DiskNotFound: No disk at vdc
2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp
2013-10-18 16:58:11.298 2478 DEBUG qpid.messaging.io.raw [-] SENT[2fa6cb0]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x0
0\x08\x00\x00\x00\x00\x00\x00\x14\xb5' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480
2013-10-18 16:58:14.265 2478 DEBUG qpid.messaging.io.raw [-] READ[2fa6cb0]: '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00' readable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:416
2013-10-18 16:58:14.266 2478 DEBUG qpid.messaging.io.ops [-] RCVD[2fa6cb0]: ConnectionHeartbeat() write /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:654
2013-10-18 16:58:14.291 2478 DEBUG qpid.messaging.io.raw [-] READ[2fca998]: '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00' readable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:416
2013-10-18 16:58:14.291 2478 DEBUG qpid.messaging.io.ops [-] RCVD[2fca998]: ConnectionHeartbeat() write /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:654
I was able to reproduce this. The issue is a subtle bug with how we handle block devices on the compute side during rebuild.
I will comment with more detail on the upstream bug as it requires upstream development work. From what I know now - it seems that the fix is not major and can be backported - however I will propose it for stable-havana upstream.
If we consider this bug to be critical enough to backport the fix before the next stable release - I don't see any major issues with that.
verified on openstack-nova-compute-2013.2-9.el6ost.noarch
we do not detach the volume when resizing and after resize we can detach and re-attach with no issues.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHEA-2013-1859.html
Created attachment 813796 [details] logs Description of problem: I had an instance with no volume attached and I took a backup of the instance after taking the backup I attached a volume and rebuild the instance from the backup. once the rebuild has finished, the volume no longer appears as attached. when we try to attach it to the same device it will give an error that the device is already in use and if we try to attach it to a different device it appears as if it works but when we try to detach the device we get DiskNotFound: No disk at vdc (which is the original device). Version-Release number of selected component (if applicable): openstack-nova-compute-2013.2-0.25.rc1.el6ost.noarch How reproducible: 100% Steps to Reproduce: 1. launch an instance from an image and take a backup of the instance 2. attach a volume to the instance 3. rebuild the instance from the backup without detaching the volume first 4. try to re-attach the volume to the instance on the same device 5. attach the volume to the instance on a different device 6. detach the volume Actual results: 1. after the rebuild the volume no longer appears as attached 2. when we try to attach to the same device, although the volume does not appear as attached, it is actually still attached in the instance 3. we can attach the volume to a different device on the same instance 4. when we try to detach it, the detach is sent to the original device and we fail to detach with No Disk error. Expected results: if we allow rebuild of instance with a volume we need to either detach completely or not detach at all Additional info: logs 2013-10-18 16:54:36.632 2478 DEBUG qpid.messaging [-] RETR[2fca830]: Message(properties={'x-amqp-0-10.routing-key': u'reply_70fb16e321724b38b3d3face4e83f363'}, content={u'oslo.message': u'{"_unique_id": "dd2a85b63c56498c8f2835f9b96e9bb9" , "failure": null, "_msg_id": "7ce524cebbb34aecab9d608a48103a1c", "result": null, "ending": true}', u'oslo.version': u'2.0'}) _get /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py:654 2013-10-18 16:54:36.633 2478 DEBUG qpid.messaging.io.ops [-] SENT[2fa6cb0]: MessageFlow(destination='0', unit=0, value=1L, id=serial(5206)) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686 2013-10-18 16:54:36.634 2478 DEBUG qpid.messaging.io.ops [-] SENT[2fa6cb0]: SessionCompleted(commands=[0-5199]) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686 2013-10-18 16:54:36.639 2478 ERROR nova.openstack.common.rpc.amqp [req-4a884bb4-5ba6-403d-8be7-df1eeebc1324 bbce236d5aac4d1dbc086a8835ed0ebc d09f3bf0f9224affa92ab97010b37270] Exception during message handling 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp **args) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp payload) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp pass 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3624, in reserve_block_device_name 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return do_reserve() 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3613, in do_reserve 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp context, instance, bdms, device) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/utils.py", line 135, in get_device_name_for_instance 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp mappings['root'], device) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/utils.py", line 217, in get_next_device_name 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp raise exception.DevicePathInUse(path=device) 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp DevicePathInUse: The supplied device path (/dev/vdc) is in use. 2013-10-18 16:54:36.639 2478 TRACE nova.openstack.common.rpc.amqp 2013-10-18 16:54:36.641 2478 ERROR nova.openstack.common.rpc.common [req-4a884bb4-5ba6-403d-8be7-df1eeebc1324 bbce236d5aac4d1dbc086a8835ed0ebc d09f3bf0f9224affa92ab97010b37270] Returning exception The supplied device path (/dev/vdc) is i n use. to caller 2013-10-18 16:54:36.642 2478 ERROR nova.openstack.common.rpc.common [req-4a884bb4-5ba6-403d-8be7-df1eeebc1324 bbce236d5aac4d1dbc086a8835ed0ebc d09f3bf0f9224affa92ab97010b37270] ['Traceback (most recent call last):\n', ' File "/usr/lib/p ython2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data\n **args)\n', ' File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyo bj, method)(ctxt, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped\n payload)\n', ' File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped\n return f(self, con text, *args, **kw)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function\n pass\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function\ n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib/python2.6/site-packages/nova/compu te/manager.py", line 258, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3624, in reserve_block_device_name\n return do_reserve() : 2013-10-18 16:58:11.289 2478 DEBUG qpid.messaging [-] RETR[2fca830]: Message(properties={'x-amqp-0-10.routing-key': u'reply_70fb16e321724b38b3d3face4e83f363'}, content={u'oslo.message': u'{"_unique_id": "20a035f66eac46f7b44e8420cd5a2d14" , "failure": null, "_msg_id": "27d7abfd907146c4a0e052cf067a4d69", "result": null, "ending": true}', u'oslo.version': u'2.0'}) _get /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py:654 2013-10-18 16:58:11.289 2478 DEBUG qpid.messaging.io.ops [-] SENT[2fa6cb0]: MessageFlow(destination='0', unit=0, value=1L, id=serial(5308)) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686 2013-10-18 16:58:11.290 2478 DEBUG qpid.messaging.io.ops [-] SENT[2fa6cb0]: SessionCompleted(commands=[0-5301]) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686 2013-10-18 16:58:11.296 2478 ERROR nova.openstack.common.rpc.amqp [req-62e56d8a-9dc1-447c-acef-2b2dafdd5079 bbce236d5aac4d1dbc086a8835ed0ebc d09f3bf0f9224affa92ab97010b37270] Exception during message handling 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp **args) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp payload) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp pass 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3760, in detach_volume 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp self._detach_volume(context, instance, bdm) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3732, in _detach_volume 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp self.volume_api.roll_detaching(context, volume_id) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3725, in _detach_volume 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp encryption=encryption) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1205, in detach_volume 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp raise exception.DiskNotFound(location=disk_dev) 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp DiskNotFound: No disk at vdc 2013-10-18 16:58:11.296 2478 TRACE nova.openstack.common.rpc.amqp 2013-10-18 16:58:11.298 2478 DEBUG qpid.messaging.io.raw [-] SENT[2fa6cb0]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x0 0\x08\x00\x00\x00\x00\x00\x00\x14\xb5' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480 2013-10-18 16:58:14.265 2478 DEBUG qpid.messaging.io.raw [-] READ[2fa6cb0]: '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00' readable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:416 2013-10-18 16:58:14.266 2478 DEBUG qpid.messaging.io.ops [-] RCVD[2fa6cb0]: ConnectionHeartbeat() write /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:654 2013-10-18 16:58:14.291 2478 DEBUG qpid.messaging.io.raw [-] READ[2fca998]: '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00' readable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:416 2013-10-18 16:58:14.291 2478 DEBUG qpid.messaging.io.ops [-] RCVD[2fca998]: ConnectionHeartbeat() write /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:654