Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1507636

Summary:	Unable to migrate in-use Cinder volume between backends.
Product:	Red Hat OpenStack	Reporter:	Rafal Szmigiel <rszmigie>
Component:	openstack-cinder	Assignee:	Jon Bernard <jobernar>
Status:	CLOSED DUPLICATE	QA Contact:	Avi Avraham <aavraham>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	11.0 (Ocata)	CC:	gcharot, rszmigie, scohen, srevivo
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-11-01 13:25:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rafal Szmigiel 2017-10-30 19:07:19 UTC

Description of problem:

Despite of what is said in OSP11 docs (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/storage_guide/ch-cinder#section-volumes-advanced-migrate):

2.4.2. Migrate a Volume
The Block Storage service allows you to migrate volumes between hosts or back ends. You can migrate a volume currently in-use (attached to an instance), but not volumes that have snapshots.

I am unable to migrate volume in-use between backends. The error I can observe on control node where openstack-cinder-volume is very similar to the one described in the following article:
https://access.redhat.com/solutions/3194652

and looks like the following:

[stack@director ~]$ openstack volume type list
+--------------------------------------+------+-----------+
| ID                                   | Name | Is Public |
+--------------------------------------+------+-----------+
| b5a939f8-da61-4758-8f43-2114bdd208dd | nfs  | True      |
| 4fa6cbb1-137e-4e9a-9817-bc356ca93765 | lvm  | True      |
+--------------------------------------+------+-----------+


[stack@director ~]$ openstack volume list
+--------------------------------------+--------------+--------+------+----------------------------------------+
| ID                                   | Display Name | Status | Size | Attached to                            |
+--------------------------------------+--------------+--------+------+----------------------------------------+
| bf4c76dd-138c-46d1-a2cd-a7ee607ea720 |              | in-use |    1 | Attached to test-instance on /dev/vda  |
+--------------------------------------+--------------+--------+------+----------------------------------------+

[stack@director ~]$ openstack volume show bf4c76dd-138c-46d1-a2cd-a7ee607ea720
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                          | Value                                                                                                                                                     |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments                    | [{u'server_id': u'76ec3056-a83a-4fc3-bb2b-5c98248d8173', u'attachment_id': u'c53f63fa-338e-4d60-a1f7-c1834798c2f0', u'attached_at':                       |
|                                | u'2017-10-30T17:35:30.000000', u'host_name': None, u'volume_id': u'bf4c76dd-138c-46d1-a2cd-a7ee607ea720', u'device': u'/dev/vda', u'id': u'bf4c76dd-138c- |
|                                | 46d1-a2cd-a7ee607ea720'}]                                                                                                                                 |
| availability_zone              | nova                                                                                                                                                      |
| bootable                       | true                                                                                                                                                      |
| consistencygroup_id            | None                                                                                                                                                      |
| created_at                     | 2017-10-30T17:35:03.000000                                                                                                                                |
| description                    |                                                                                                                                                           |
| encrypted                      | False                                                                                                                                                     |
| id                             | bf4c76dd-138c-46d1-a2cd-a7ee607ea720                                                                                                                      |
| migration_status               | error                                                                                                                                                     |
| multiattach                    | False                                                                                                                                                     |
| name                           |                                                                                                                                                           |
| os-vol-host-attr:host          | hostgroup@tripleo_nfs#tripleo_nfs                                                                                                                         |
| os-vol-mig-status-attr:migstat | error                                                                                                                                                     |
| os-vol-mig-status-attr:name_id | None                                                                                                                                                      |
| os-vol-tenant-attr:tenant_id   | a0eb7226b50744ab98cbae45f0de1b60                                                                                                                          |
| properties                     | attached_mode='rw'                                                                                                                                        |
| replication_status             | None                                                                                                                                                      |
| size                           | 1                                                                                                                                                         |
| snapshot_id                    | None                                                                                                                                                      |
| source_volid                   | None                                                                                                                                                      |
| status                         | in-use                                                                                                                                                    |
| type                           | None                                                                                                                                                      |
| updated_at                     | 2017-10-30T18:44:26.000000                                                                                                                                |
| user_id                        | e58f062056f54ae08d6731546d8cd389                                                                                                                          |
| volume_image_metadata          | {u'container_format': u'bare', u'min_ram': u'128', u'disk_format': u'qcow2', u'image_name': u'Cirros-0.3.5', u'image_id': u'3d14111e-8946-4deb-846a-      |
|                                | 2ff79e719540', u'checksum': u'f8ab98ff5e73ebab884d80c9dc9c7290', u'min_disk': u'1', u'size': u'13267968'}                                                 |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+


[stack@director ~]$ cinder retype --migration-policy on-demand bf4c76dd-138c-46d1-a2cd-a7ee607ea720 lvm



2017-10-30 19:02:05.266 315252 INFO cinder.volume.flows.manager.create_volume [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Volume e09e3fb8-259b-4f19-8cf2-a9fc0161eed1: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-e09e3fb8-259b-4f19-8cf2-a9fc0161eed1'}
2017-10-30 19:02:05.795 315252 INFO cinder.volume.flows.manager.create_volume [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Volume volume-e09e3fb8-259b-4f19-8cf2-a9fc0161eed1 (e09e3fb8-259b-4f19-8cf2-a9fc0161eed1): created successfully
2017-10-30 19:02:05.800 315252 INFO cinder.volume.manager [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Created volume successfully.
2017-10-30 19:02:06.343 315246 WARNING keystoneauth.identity.generic.base [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Failed to copy volume bf4c76dd-138c-46d1-a2cd-a7ee607ea720 to e09e3fb8-259b-4f19-8cf2-a9fc0161eed1
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager Traceback (most recent call last):
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1844, in _migrate_volume_generic
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     new_volume.id)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/compute/nova.py", line 250, in update_server_volume
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     new_volume_id)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 68, in update_server_volume
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     body, "volumeAttachment")
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 375, in _update
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     resp, body = self.api.client.put(url, body=body)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 226, in put
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     return self.request(url, 'PUT', **kwargs)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 74, in request
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     **kwargs)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 374, in request
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     return self.session.request(url, method, **kwargs)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     return wrapped(*args, **kwargs)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 491, in request
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     auth_headers = self.get_auth_headers(auth)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     return auth.get_headers(self, **kwargs)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     token = self.get_token(session)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     return self.get_access(session).auth_token
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     self.auth_ref = self.get_auth_ref(session)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 196, in get_auth_ref
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     self._plugin = self._do_create_plugin(session)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager DiscoveryFailure: Could not determine a suitable URL for the plugin
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager 
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Exception during message handling
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2401, in retype
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     new_reservations, status_update)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2397, in retype
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     new_type_id=new_type_id)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2039, in migrate_volume
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     volume.save()
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2032, in migrate_volume
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1851, in _migrate_volume_generic
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     new_volume)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1844, in _migrate_volume_generic
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     new_volume.id)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/compute/nova.py", line 250, in update_server_volume
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     new_volume_id)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 68, in update_server_volume
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     body, "volumeAttachment")
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 375, in _update
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.put(url, body=body)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 226, in put
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     return self.request(url, 'PUT', **kwargs)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 74, in request
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     **kwargs)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 374, in request
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     return self.session.request(url, method, **kwargs)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     return wrapped(*args, **kwargs)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 491, in request
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     auth_headers = self.get_auth_headers(auth)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     return auth.get_headers(self, **kwargs)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     token = self.get_token(session)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     return self.get_access(session).auth_token
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     self.auth_ref = self.get_auth_ref(session)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 196, in get_auth_ref
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     self._plugin = self._do_create_plugin(session)
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server     raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server DiscoveryFailure: Could not determine a suitable URL for the plugin
2017-10-30 19:02:06.455 315246 ERROR oslo_messaging.rpc.server 
2017-10-30 19:02:06.492 315252 INFO cinder.volume.targets.iscsi [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Skipping remove_export. No iscsi_target is presently exported for volume: e09e3fb8-259b-4f19-8cf2-a9fc0161eed1
2017-10-30 19:02:07.368 315252 INFO cinder.volume.utils [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Performing secure delete on volume: /dev/mapper/cinder--lvm-volume--e09e3fb8--259b--4f19--8cf2--a9fc0161eed1
2017-10-30 19:02:08.644 315252 INFO cinder.volume.utils [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Volume copy 1024.00 MB at 1024.00 MB/s
2017-10-30 19:02:08.992 315252 INFO cinder.volume.drivers.lvm [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Successfully deleted volume: e09e3fb8-259b-4f19-8cf2-a9fc0161eed1
2017-10-30 19:02:09.014 315252 INFO cinder.volume.manager [req-d21f7d19-f7e0-4dde-b7b9-6929609fb8e9 e58f062056f54ae08d6731546d8cd389 a0eb7226b50744ab98cbae45f0de1b60 - default default] Deleted volume successfully.



Version-Release number of selected component (if applicable):
[root@overcloud-controller-0 cinder]# rpm -qa | grep openstack
openstack-gnocchi-indexer-sqlalchemy-3.1.10-2.el7ost.noarch
openstack-nova-novncproxy-15.0.6-6.el7ost.noarch
openstack-octavia-api-0.10.0-9.el7ost.noarch
openstack-zaqar-4.0.0-3.el7ost.noarch
openstack-gnocchi-metricd-3.1.10-2.el7ost.noarch
openstack-swift-proxy-2.13.1-1.el7ost.noarch
openstack-swift-container-2.13.1-1.el7ost.noarch
openstack-heat-common-8.0.3-1.el7ost.noarch
openstack-sahara-engine-6.0.0-4.el7ost.noarch
openstack-ceilometer-polling-8.1.0-2.el7ost.noarch
openstack-panko-api-2.0.2-1.el7ost.noarch
openstack-aodh-notifier-4.0.1-2.el7ost.noarch
openstack-neutron-metering-agent-10.0.2-8.el7ost.noarch
openstack-swift-account-2.13.1-1.el7ost.noarch
openstack-puppet-modules-10.0.0-1.el7ost.noarch
openstack-dashboard-theme-11.0.3-3.el7ost.noarch
openstack-mistral-common-4.0.2-1.el7ost.noarch
openstack-ceilometer-api-8.1.0-2.el7ost.noarch
openstack-mistral-api-4.0.2-1.el7ost.noarch
openstack-ironic-conductor-7.0.2-2.el7ost.noarch
openstack-sahara-common-6.0.0-4.el7ost.noarch
python-django-openstack-auth-3.1.1-2.el7ost.noarch
openstack-aodh-common-4.0.1-2.el7ost.noarch
openstack-cinder-10.0.4-3.el7ost.noarch
openstack-nova-cert-15.0.6-6.el7ost.noarch
openstack-glance-14.0.0-3.el7ost.noarch
openstack-keystone-11.0.3-1.el7ost.noarch
openstack-neutron-openvswitch-10.0.2-8.el7ost.noarch
openstack-heat-engine-8.0.3-1.el7ost.noarch
openstack-swift-object-2.13.1-1.el7ost.noarch
openstack-neutron-ml2-10.0.2-8.el7ost.noarch
openstack-panko-common-2.0.2-1.el7ost.noarch
openstack-gnocchi-common-3.1.10-2.el7ost.noarch
openstack-ceilometer-central-8.1.0-2.el7ost.noarch
openstack-nova-conductor-15.0.6-6.el7ost.noarch
openstack-aodh-evaluator-4.0.1-2.el7ost.noarch
openstack-octavia-worker-0.10.0-9.el7ost.noarch
openstack-sahara-ui-6.0.0-3.el7ost.noarch
openstack-heat-api-cfn-8.0.3-1.el7ost.noarch
python-openstacksdk-0.9.13-1.el7ost.noarch
openstack-dashboard-11.0.3-3.el7ost.noarch
openstack-octavia-common-0.10.0-9.el7ost.noarch
openstack-nova-compute-15.0.6-6.el7ost.noarch
openstack-gnocchi-statsd-3.1.10-2.el7ost.noarch
openstack-ec2-api-4.0.0-3.el7ost.noarch
openstack-nova-placement-api-15.0.6-6.el7ost.noarch
openstack-mistral-engine-4.0.2-1.el7ost.noarch
openstack-neutron-lbaas-10.0.1-2.el7ost.noarch
openstack-manila-4.0.0-6.el7ost.noarch
openstack-selinux-0.8.9-0.1.el7ost.noarch
puppet-openstacklib-10.4.1-1.el7ost.noarch
puppet-openstack_extras-10.3.1-1.el7ost.noarch
openstack-neutron-common-10.0.2-8.el7ost.noarch
openstack-sahara-api-6.0.0-4.el7ost.noarch
openstack-ceilometer-common-8.1.0-2.el7ost.noarch
openstack-ceilometer-collector-8.1.0-2.el7ost.noarch
openstack-nova-scheduler-15.0.6-6.el7ost.noarch
openstack-aodh-listener-4.0.1-2.el7ost.noarch
openstack-sahara-6.0.0-4.el7ost.noarch
openstack-neutron-sriov-nic-agent-10.0.2-8.el7ost.noarch
openstack-heat-api-8.0.3-1.el7ost.noarch
openstack-swift-plugin-swift3-1.11.0-8.el7ost.noarch
openstack-ironic-common-7.0.2-2.el7ost.noarch
python-openstack-mistral-4.0.2-1.el7ost.noarch
openstack-ceilometer-notification-8.1.0-2.el7ost.noarch
openstack-ceilometer-compute-8.1.0-2.el7ost.noarch
openstack-aodh-api-4.0.1-2.el7ost.noarch
openstack-ironic-api-7.0.2-2.el7ost.noarch
python-openstackclient-3.8.1-1.el7ost.noarch
openstack-nova-migration-15.0.6-6.el7ost.noarch
openstack-mistral-executor-4.0.2-1.el7ost.noarch
openstack-manila-share-4.0.0-6.el7ost.noarch
openstack-neutron-10.0.2-8.el7ost.noarch
openstack-nova-console-15.0.6-6.el7ost.noarch
openstack-octavia-health-manager-0.10.0-9.el7ost.noarch
openstack-heat-api-cloudwatch-8.0.3-1.el7ost.noarch
openstack-nova-common-15.0.6-6.el7ost.noarch
openstack-gnocchi-api-3.1.10-2.el7ost.noarch
openstack-nova-api-15.0.6-6.el7ost.noarch
openstack-octavia-housekeeping-0.10.0-9.el7ost.noarch
openstack-manila-ui-2.7.1-4.el7ost.noarch



How reproducible:
All the time.

Steps to Reproduce:
1. Create instance booted from volume.
2. Attempt to change volume type either via CLI or Horizon as described here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/storage_guide/ch-cinder#section-volumes-advanced-migrate
3. Observe errors from openstack-cinder-volume service.

Actual results:
Migration of attached volume between backends fails.

Expected results:
Migration of attached volume between backends succeed.


Additional info:

Comment 1 Eric Harney 2017-10-30 19:14:25 UTC

This failure message indicates that Cinder can't find the Nova endpoint in the keystone catalog:

2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1844, in _migrate_volume_generic
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     new_volume.id)
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/compute/nova.py", line 250, in update_server_volume
...
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager     raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
2017-10-30 19:02:06.343 315246 ERROR cinder.volume.manager DiscoveryFailure: Could not determine a suitable URL for the plugin

Could you post your cinder.conf and output of keystone endpoint-list and service-list?

Comment 2 Rafal Szmigiel 2017-10-30 19:19:43 UTC

[root@overcloud-controller-0 heat-admin]# cat /etc/cinder/cinder.conf  | grep -v -E '^#|^$'
[DEFAULT]
glance_api_servers = http://172.17.0.14:9292
glance_api_version = 2
enable_v3_api = True
host = hostgroup
storage_availability_zone = nova
default_availability_zone = nova
enabled_backends = tripleo_nfs, tripleo_lvm
nova_catalog_info = compute:nova:internalURL
nova_catalog_admin_info = compute:nova:adminURL
scheduler_driver = cinder.scheduler.filter_scheduler.FilterScheduler
osapi_volume_listen = overcloud-controller-0.internalapi.localdomain
osapi_volume_workers = 4
log_dir = /var/log/cinder
rpc_backend = rabbit
control_exchange = openstack
api_paste_config = /etc/cinder/api-paste.ini
[backend]
[barbican]
[brcd_fabric_example]
[cisco_fabric_example]
[coordination]
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://cinder:y4gBH3nbpVyQxEbvEeYhFuDZn.0.14/cinder?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo
max_retries = -1
db_max_retries = -1
[fc-zone-manager]
[healthcheck]
[key_manager]
[keystone_authtoken]
auth_uri = http://172.17.0.14:5000/v2.0
auth_type = password
auth_url=http://172.17.0.14:5000
username=cinder
password=***SNIP***
project_name=service
[matchmaker_redis]
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
driver =messagingv2
[oslo_messaging_rabbit]
rabbit_hosts = overcloud-controller-0.internalapi.localdomain:5672,overcloud-controller-1.internalapi.localdomain:5672,overcloud-controller-2.internalapi.localdomain:5672
rabbit_use_ssl = False
rabbit_userid = guest
rabbit_password = ***SNIP***
rabbit_ha_queues = True
heartbeat_timeout_threshold = 60
[oslo_messaging_zmq]
[oslo_middleware]
enable_proxy_headers_parsing = True
[oslo_policy]
policy_file = /etc/cinder/policy.json
[oslo_reports]
[oslo_versionedobjects]
[profiler]
[ssl]
[tripleo_nfs]
volume_backend_name=tripleo_nfs
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config=/etc/cinder/shares-nfs.conf
nfs_mount_options=rw,sync
nas_secure_file_operations=False
nas_secure_file_permissions=False
[tripleo_lvm]
volume_backend_name=tripleo_lvm
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
iscsi_protocol = iscsi
volume_group = cinder-lvm


[stack@director ~]$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+
| ID                               | Region    | Service Name | Service Type   |
+----------------------------------+-----------+--------------+----------------+
| 43722db55d70465086517c0f5e1019e8 | regionOne | keystone     | identity       |
| 0cafa30b6092425cbed5a2968fff9a6a | regionOne | cinderv3     | volumev3       |
| 9d7e1ae7be8844dfb21f86a02589a20b | regionOne | glance       | image          |
| aebb20d50ece49999ca8d7354d7ecb55 | regionOne | cinderv2     | volumev2       |
| 50e2f1ec2f4a462cae6e3d141e1b1e07 | regionOne | ceilometer   | metering       |
| 3dd1885cd56e4ef9a5f3882a8d0f55d6 | regionOne | heat-cfn     | cloudformation |
| b9ef3fd454b24462a4aee9e4d45c975c | regionOne | aodh         | alarming       |
| 6f03434333e54d2982d818aa8dc1db03 | regionOne | neutron      | network        |
| 50a1fe3dce9540a5aa4073de16f4033e | regionOne | gnocchi      | metric         |
| e5abb6ce475e4fccb0bd4f4b518ec126 | regionOne | swift        | object-store   |
| 0e138b1447d841cab5e77e278c8009f2 | regionOne | panko        | event          |
| defb68c088ac42e3bf243c99df6ce980 | regionOne | placement    | placement      |
| cdc285291aa643719839a2cc3e366ea1 | regionOne | heat         | orchestration  |
| 20f30da3bd344cda9025968be5841341 | regionOne | nova         | compute        |
| 3debbf71777042feae9c562e657234d1 | regionOne | cinder       | volume         |
+----------------------------------+-----------+--------------+----------------+

[stack@director ~]$ openstack service list --long
+----------------------------------+------------+----------------+----------------------------------+
| ID                               | Name       | Type           | Description                      |
+----------------------------------+------------+----------------+----------------------------------+
| 020fcc3621cc445ba1c36cad80e4b6bf | swift      | object-store   | Openstack Object-Store Service   |
| 1fe8648a8cf44b6fa43fe0cbe7c035d5 | cinderv2   | volumev2       | Cinder Service v2                |
| 22d29fd680ea488f93d185227f3fc41f | cinderv3   | volumev3       | Cinder Service v3                |
| 4de2743ef2b64232952d27b2e6972a03 | panko      | event          | OpenStack Event Service          |
| 4ebae1cdf1264689adad4aca4ddd4a7b | placement  | placement      | Openstack Placement Service      |
| 517e89182df647d8894463821bda385c | glance     | image          | OpenStack Image Service          |
| 59830bb949a645c3b403a08c53a2b52f | gnocchi    | metric         | OpenStack Metric Service         |
| 5af45943e95d48c0926aebee91e89567 | ceilometer | metering       | Openstack Metering Service       |
| 685d61f80966451ebdfe7385c31fab4d | heat-cfn   | cloudformation | Openstack Cloudformation Service |
| 9cbc219abcb64a78a7ab9b7a2ece6e71 | neutron    | network        | Neutron Networking Service       |
| 9cf06e58e2b84bcfa3e423bef37a7326 | heat       | orchestration  | Openstack Orchestration Service  |
| a25f0c03c6de4a0487d1681a221d009f | keystone   | identity       | OpenStack Identity Service       |
| db6c716d1d2a4874af8dcc1e3515a40d | aodh       | alarming       | OpenStack Alarming Service       |
| e70b82f5fe824239a63c27174be27504 | cinder     | volume         | Cinder Service                   |
| f841883b8b5b4e5b8793ef9941cdcff6 | nova       | compute        | Openstack Compute Service        |
+----------------------------------+------------+----------------+----------------------------------+

Comment 3 Eric Harney 2017-10-30 19:37:24 UTC

Can you also include "openstack endpoint show nova" ?

Thanks,
Eric

Comment 4 Rafal Szmigiel 2017-10-30 19:43:06 UTC

[stack@director ~]$ openstack endpoint show nova
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| adminurl     | http://172.17.0.14:8774/v2.1     |
| enabled      | True                             |
| id           | 20f30da3bd344cda9025968be5841341 |
| internalurl  | http://172.17.0.14:8774/v2.1     |
| publicurl    | http://192.168.122.10:8774/v2.1  |
| region       | regionOne                        |
| service_id   | f841883b8b5b4e5b8793ef9941cdcff6 |
| service_name | nova                             |
| service_type | compute                          |
+--------------+----------------------------------+