Bug 1328476

Summary: cinder migrate fail
Product: Red Hat OpenStack Reporter: Pratik Pravin Bandarkar <pbandark>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED NOTABUG QA Contact: nlevinki <nlevinki>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: egafford, eharney, srevivo
Target Milestone: asyncKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-09 19:41:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Pratik Pravin Bandarkar 2016-04-19 13:28:13 UTC
Description of problem:
if you update "nova_catalog_info" & "nova_catalog_admin_info" value than default and further if you revert it back to original values, `cinder migrate` for attached volume will fail.


Version-Release number of selected component (if applicable):
RHOS8

How reproducible:
100%

Steps to Reproduce:

- update cinder.conf:
[...]
nova_catalog_info = compute:Compute Service:publicURL
nova_catalog_admin_info = compute:Compute Service:adminURL
[...]

- Restart cinder* services.

- reset the values to default:
[...]
nova_catalog_info = compute:nova:publicURL
nova_catalog_admin_info = compute:nova:adminURL
[...]

- restart cinder* services.

- attach volume to the instance and try to migrate it in between backend:
[root@dhcp200-223 ~(keystone_admin)]# nova volume-attach 884b8e00-c6ad-4523-ae01-c48110202c37 f909df6f-f82f-4ba0-8fdd-7287eca23564
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | f909df6f-f82f-4ba0-8fdd-7287eca23564 |
| serverId | 884b8e00-c6ad-4523-ae01-c48110202c37 |
| volumeId | f909df6f-f82f-4ba0-8fdd-7287eca23564 |
+----------+--------------------------------------+

[root@dhcp200-223 ~(keystone_admin)]# cinder show f909df6f-f82f-4ba0-8fdd-7287eca23564
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                Property               |                                                                                                                                  Value                                                                                                                                  |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|              attachments              | [{u'server_id': u'884b8e00-c6ad-4523-ae01-c48110202c37', u'attachment_id': u'0c9b1f5e-32a2-4f30-b472-0fe7e7f1eb6b', u'host_name': None, u'volume_id': u'f909df6f-f82f-4ba0-8fdd-7287eca23564', u'device': u'/dev/vdb', u'id': u'f909df6f-f82f-4ba0-8fdd-7287eca23564'}] |
|           availability_zone           |                                                                                                                                   nova                                                                                                                                  |
|                bootable               |                                                                                                                                  false                                                                                                                                  |
|          consistencygroup_id          |                                                                                                                                   None                                                                                                                                  |
|               created_at              |                                                                                                                        2016-04-19T20:40:40.000000                                                                                                                       |
|              description              |                                                                                                                                   None                                                                                                                                  |
|               encrypted               |                                                                                                                                  False                                                                                                                                  |
|                   id                  |                                                                                                                   f909df6f-f82f-4ba0-8fdd-7287eca23564                                                                                                                  |
|                metadata               |                                                                                                             {u'readonly': u'False', u'attached_mode': u'rw'}                                                                                                            |
|            migration_status           |                                                                                                                                 success                                                                                                                                 |
|              multiattach              |                                                                                                                                  False                                                                                                                                  |
|                  name                 |                                                                                                                                   None                                                                                                                                  |
|         os-vol-host-attr:host         |                                                                                                                      dhcp200-223@production#nfs123                                                                                                                      |
|     os-vol-mig-status-attr:migstat    |                                                                                                                                 success                                                                                                                                 |
|     os-vol-mig-status-attr:name_id    |                                                                                                                   074d5262-fb2e-490a-8fe7-ab049b24e7fa                                                                                                                  |
|      os-vol-tenant-attr:tenant_id     |                                                                                                                     cb5f30f3737a4dc1ac766d3c68002cb9                                                                                                                    |
|   os-volume-replication:driver_data   |                                                                                                                                   None                                                                                                                                  |
| os-volume-replication:extended_status |                                                                                                                                   None                                                                                                                                  |
|           replication_status          |                                                                                                                                 disabled                                                                                                                                |
|                  size                 |                                                                                                                                    1                                                                                                                                    |
|              snapshot_id              |                                                                                                                                   None                                                                                                                                  |
|              source_volid             |                                                                                                                                   None                                                                                                                                  |
|                 status                |                                                                                                                                  in-use                                                                                                                                 |
|                user_id                |                                                                                                                     9fbade688f294569b040a223495c7825                                                                                                                    |
|              volume_type              |                                                                                                                                   None                                                                                                                                  |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


[root@dhcp200-223 ~(keystone_admin)]# cinder migrate f909df6f-f82f-4ba0-8fdd-7287eca23564 dhcp200-223@development#nfs123
Request to migrate volume <Volume: f909df6f-f82f-4ba0-8fdd-7287eca23564> has been accepted.

[root@dhcp200-223 ~(keystone_admin)]# cinder show f909df6f-f82f-4ba0-8fdd-7287eca23564
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                Property               |                                                                                                                                  Value                                                                                                                                  |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|              attachments              | [{u'server_id': u'884b8e00-c6ad-4523-ae01-c48110202c37', u'attachment_id': u'0c9b1f5e-32a2-4f30-b472-0fe7e7f1eb6b', u'host_name': None, u'volume_id': u'f909df6f-f82f-4ba0-8fdd-7287eca23564', u'device': u'/dev/vdb', u'id': u'f909df6f-f82f-4ba0-8fdd-7287eca23564'}] |
|           availability_zone           |                                                                                                                                   nova                                                                                                                                  |
|                bootable               |                                                                                                                                  false                                                                                                                                  |
|          consistencygroup_id          |                                                                                                                                   None                                                                                                                                  |
|               created_at              |                                                                                                                        2016-04-19T20:40:40.000000                                                                                                                       |
|              description              |                                                                                                                                   None                                                                                                                                  |
|               encrypted               |                                                                                                                                  False                                                                                                                                  |
|                   id                  |                                                                                                                   f909df6f-f82f-4ba0-8fdd-7287eca23564                                                                                                                  |
|                metadata               |                                                                                                             {u'readonly': u'False', u'attached_mode': u'rw'}                                                                                                            |
|            migration_status           |                                                                                                                                  error                                                                                                                                  |
|              multiattach              |                                                                                                                                  False                                                                                                                                  |
|                  name                 |                                                                                                                                   None                                                                                                                                  |
|         os-vol-host-attr:host         |                                                                                                                      dhcp200-223@production#nfs123                                                                                                                      |
|     os-vol-mig-status-attr:migstat    |                                                                                                                                  error                                                                                                                                  |
|     os-vol-mig-status-attr:name_id    |                                                                                                                   074d5262-fb2e-490a-8fe7-ab049b24e7fa                                                                                                                  |
|      os-vol-tenant-attr:tenant_id     |                                                                                                                     cb5f30f3737a4dc1ac766d3c68002cb9                                                                                                                    |
|   os-volume-replication:driver_data   |                                                                                                                                   None                                                                                                                                  |
| os-volume-replication:extended_status |                                                                                                                                   None                                                                                                                                  |
|           replication_status          |                                                                                                                                 disabled                                                                                                                                |
|                  size                 |                                                                                                                                    1                                                                                                                                    |
|              snapshot_id              |                                                                                                                                   None                                                                                                                                  |
|              source_volid             |                                                                                                                                   None                                                                                                                                  |
|                 status                |                                                                                                                                  in-use                                                                                                                                 |
|                user_id                |                                                                                                                     9fbade688f294569b040a223495c7825                                                                                                                    |
|              volume_type              |                                                                                                                                   None                                                                                                                                  |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

___________________________________________


volume.log:
[...]
2016-04-19 16:53:30.354 25145 INFO cinder.volume.manager [req-34944883-5cac-49b6-9cfd-15ef3c75259c 9fbade688f294569b040a223495c7825 cb5f30f3737a4dc1ac766d3c68002cb9 - - -] Migrate volume completed successfully.
2016-04-19 16:53:31.556 25157 INFO cinder.volume.manager [req-5ef2b962-1728-44ad-97be-053d92f3fe52 9fbade688f294569b040a223495c7825 cb5f30f3737a4dc1ac766d3c68002cb9 - - -] Initialize volume connection completed successfully.
2016-04-19 16:53:32.200 25157 INFO cinder.volume.manager [req-c7d8c380-2be0-492c-8f58-594df8c71b0b 9fbade688f294569b040a223495c7825 cb5f30f3737a4dc1ac766d3c68002cb9 - - -] Terminate volume connection completed successfully.
2016-04-19 16:53:32.402 25145 DEBUG cinder.volume.manager [req-2971f018-087b-44fe-b054-ad8433ed520c 9fbade688f294569b040a223495c7825 cb5f30f3737a4dc1ac766d3c68002cb9 - - -] migrate_volume_completion: completing migration for volume f909df6f-f82f-4ba0-8fdd-7287eca23564 (temporary volume 8dd5f90a-d81b-4779-955b-f9bac1b54c09 migrate_volume_completion /usr/lib/python2.7/site-packages/cinder/volume/manager.py:1713
2016-04-19 16:53:32.513 25145 INFO cinder.volume.manager [req-2971f018-087b-44fe-b054-ad8433ed520c 9fbade688f294569b040a223495c7825 cb5f30f3737a4dc1ac766d3c68002cb9 - - -] migrate_volume_completion is cleaning up an error for volume f909df6f-f82f-4ba0-8fdd-7287eca23564 (temporary volume 8dd5f90a-d81b-4779-955b-f9bac1b54c09


Actual results:
volume migration fail.

Expected results:
volume migration should be successful.


Additional info:
prior updating cinder.conf, volume migration was successful:


[root@dhcp200-223 ~(keystone_admin)]# nova volume-attach 884b8e00-c6ad-4523-ae01-c48110202c37 71aef807-0813-4feb-af52-084141b04441
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 71aef807-0813-4feb-af52-084141b04441 |
| serverId | 884b8e00-c6ad-4523-ae01-c48110202c37 |
| volumeId | 71aef807-0813-4feb-af52-084141b04441 |
+----------+--------------------------------------+


[root@dhcp200-223 ~(keystone_admin)]# cinder migrate 71aef807-0813-4feb-af52-084141b04441 dhcp200-223@development#nfs123
Request to migrate volume <Volume: 71aef807-0813-4feb-af52-084141b04441> has been accepted.

[root@dhcp200-223 ~(keystone_admin)]# cinder show 71aef807-0813-4feb-af52-084141b04441

[...]
|              attachments              | [{u'server_id': u'884b8e00-c6ad-4523-ae01-c48110202c37', u'attachment_id': u'ccf1cb5b-9914-4aa5-b86c-8348248e432f', u'host_name': None, u'volume_id': u'71aef807-0813-4feb-af52-084141b04441', u'device': u'/dev/vdb', u'id': u'71aef807-0813-4feb-af52-084141b04441'}] |

|            migration_status           |          migrating    

|         os-vol-host-attr:host         |        dhcp200-223@production#nfs123                  



____

[root@dhcp200-223 ~(keystone_admin)]# cinder show 71aef807-0813-4feb-af52-084141b04441

[...]
|              attachments              | [{u'server_id': u'884b8e00-c6ad-4523-ae01-c48110202c37', u'attachment_id': u'f151fc79-ad13-402c-a4bb-7b55f55243dd', u'host_name': None, u'volume_id': u'71aef807-0813-4feb-af52-084141b04441', u'device': u'/dev/vdb', u'id': u'71aef807-0813-4feb-af52-084141b04441'}] |

|            migration_status           |        success    

|         os-vol-host-attr:host         |   dhcp200-223@development#nfs123