Bug 1080543

Summary: volume migration from netapp back end to LVM back end failed
Product: [Community] RDO Reporter: Yogev Rabl <yrabl>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED UPSTREAM QA Contact: Dafna Ron <dron>
Severity: high Docs Contact:
Priority: urgent    
Version: unspecifiedCC: eharney, lars, yeylon, yrabl
Target Milestone: RC   
Target Release: Icehouse   
Hardware: All   
OS: All   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-04-01 15:35:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yogev Rabl 2014-03-25 16:17:47 UTC
Description of problem:
On an AIO installation, the cinder is configured with multiple back ends: Netapp & LVM. 
A volume, 10 GB size, was created on the Netapp back end, (the back end was picked by the cinder-scheduler, not by the user). 

I ran the migration command:
# cinder migrate <volume-id> <hostname>@lvm 

after the migration failed two volumes appear in the cinder volume list:
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 405406a0-21e9-4a7b-baff-daaceed191fa | available |     None     |  10  |     None    |  false   |             |
| 4499979e-c326-4150-b3d6-2cbbbf9642c6 | available |     None     |  10  |     None    |  false   |             |
| fabdfd8b-c6e7-4f2f-a93e-60cd4556ed57 | available |     None     |  1   |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

but there's only one volume with 10 GB, and that's the one in the Netapp storage, there's no volume in Cinder's LVM back end.

The migration failed with the error (from volume.log): 
2014-03-25 17:54:12.355 6722 ERROR cinder.openstack.common.rpc.amqp [req-a58b365a-8e1f-434b-b5b3-dc33014ea12c None] Exception during message handling
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     **args)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1070, in migrate_volume
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     self.db.volume_update(ctxt, volume_ref['id'], updates)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1061, in migrate_volume
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     new_type_id)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 946, in _migrate_volume_generic
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     new_volume['migration_status'] = None
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 926, in _migrate_volume_generic
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     remote='dest')
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 304, in copy_volume_data
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     {'status': dest_orig_status})
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 298, in copy_volume_data
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     remote=dest_remote)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 382, in _attach_volume
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     rpcapi.create_export(context, volume)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 211, in create_export
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     version='1.13')
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/proxy.py", line 125, in call
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     result = rpc.call(context, real_topic, msg, timeout)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/__init__.py", line 137, in call
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     return _get_impl().call(CONF, context, topic, msg, timeout)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/impl_qpid.py", line 782, in call
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     rpc_amqp.get_connection_pool(conf, Connection))
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py", line 575, in call
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     rv = list(rv)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py", line 540, in __iter__
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     raise result
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp AttributeError: No such RPC function 'create_export'
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp
014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     **args)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp 
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 176, in dispatch
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp     raise AttributeError("No such RPC function '%s'" % method)
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp 
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp AttributeError: No such RPC function 'create_export'
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp 
2014-03-25 17:54:12.355 6722 TRACE cinder.openstack.common.rpc.amqp

Version-Release number of selected component (if applicable):
python-cinderclient-1.0.7-2.fc21.noarch
openstack-cinder-2014.1-0.6.b3.fc21.noarch
python-cinder-2014.1-0.6.b3.fc21.noarch

How reproducible:
100%

Steps to Reproduce:
1. create a volume on the Netapp back end (without associate the volume with a type to the back end).
2. migrate the volume to the LVM back end.
3.

Actual results:
The migration failed and there's a false data in the cinder volume list.

Expected results:
the migration should succeed - there should be 1 volume in the destination back end.

Additional info:

Comment 1 Lars Kellogg-Stedman 2015-04-01 15:35:40 UTC
The upstream bug was fixed back in March 2014.