Bug 1630771

Summary: Nova Live Migration Failed to Update Database
Product: Red Hat OpenStack Reporter: Md Nadeem <mnadeem>
Component: openstack-novaAssignee: OSP DFG:Compute <osp-dfg-compute>
Status: CLOSED DUPLICATE QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: low Docs Contact:
Priority: low    
Version: 10.0 (Newton)CC: asoni, berrange, dasmith, dhill, eglynn, fahmed, jhakimra, jwaterwo, kchamart, lyarwood, mbooth, panbalag, sbauza, sferdjao, sgordon, srevivo, vaggarwa, vromanso
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1347461 Environment:
Last Closed: 2018-10-10 19:13:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1347461    
Bug Blocks:    

Description Md Nadeem 2018-09-19 08:52:38 UTC
Instance physically migrated to destination compute node, we have confirmed the same via "virsh list" and instance is running and accessible.

However the "nova show <id>" still shows old source compute node as host and hypervisor_hostname of the instance. It seems the nova db has not updated with the new target compute node.

From the logs, it seems the instance migration failed at post migration steps, If this is true then it should not migrate instance to destination host. Please find below logs.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2018-09-12 15:51:36.161 683974 INFO nova.virt.libvirt.driver [req-0d32334f-4fc3-48e0-82f2-277d6e528ba0 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - - -] [instance: 985cb1a2-bf8b-4f2b-9474-a8f17dbba511] Migration operation has completed


2018-09-12 15:51:36.161 683974 INFO nova.compute.manager [req-0d32334f-4fc3-48e0-82f2-277d6e528ba0 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - - -] [instance: 985cb1a2-bf8b-4f2b-9474-a8f17dbba511] _post_live_migration() is started..
2018-09-12 15:51:36.253 683974 ERROR nova.volume.cinder [req-0d32334f-4fc3-48e0-82f2-277d6e528ba0 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - - -] Initialize connection failed for volume 52c2cc27-0e4d-4ce3-be95-094c239b7107 on host XX-comp-13.stack.enoc-airtel.com. Error: The request you have made requires authentication. (HTTP 401) Code: 401. Attempting to terminate connection.
2018-09-12 15:51:36.258 683974 ERROR nova.volume.cinder [req-0d32334f-4fc3-48e0-82f2-277d6e528ba0 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - - -] Connection between volume 52c2cc27-0e4d-4ce3-be95-094c239b7107 and host XX-comp-13.stack.enoc-airtel.com might have succeeded, but attempt to terminate connection has failed. Validate the connection and determine if manual cleanup is needed. Error: The request you have made requires authentication. (HTTP 401) Code: 401.


2018-09-12 15:51:36.299 683974 ERROR nova.compute.manager [req-0d32334f-4fc3-48e0-82f2-277d6e528ba0 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - - -] [instance: 985cb1a2-bf8b-4f2b-9474-a8f17dbba511] Live migration failed.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Comment 1 Matthew Booth 2018-09-21 11:54:45 UTC
We update instance.host in post_live_migration_at_destination which is called via rpc from _post_live_migration on the source after a lot of cleanup has already happened. I wonder why we don't update instance.host from the source at the start of that method.

Comment 2 Matthew Booth 2018-09-21 15:12:31 UTC
Looks like a valid bug, and the code is similar between master and OSP10 so a backport looks likely possible at this stage.

Comment 3 Artom Lifshitz 2018-10-10 19:13:47 UTC
Closing as duplicate of bz 1636102. All further updates will be there.

*** This bug has been marked as a duplicate of bug 1636102 ***