Bug 1942000

Summary:	VM instance live migration fails
Product:	Red Hat OpenStack	Reporter:	Eduardo Olivares <eolivare>
Component:	openstack-nova	Assignee:	OSP DFG:Compute <osp-dfg-compute>
Status:	CLOSED DUPLICATE	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	16.2 (Train)	CC:	ccamposr, dasmith, eglynn, jhakimra, kchamart, mschuppe, rsafrono, sbauza, sgordon, vromanso
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-03-23 12:00:21 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eduardo Olivares 2021-03-23 11:47:23 UTC

Description of problem:
Several tempest tests failed during VM live migration (cold migration tests pass):
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv4-geneve/101/testReport/

For example, the test test_server_connectivity_live_migration fails because migration is not performed:
AssertionError: 'compute-0.redhat.local' == 'compute-0.redhat.local' : Server did not migrate

compute-0's nova-compute logs show the following error:
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv4-geneve/101/compute-0/var/log/containers/nova/nova-compute.log.1.gz
2021-03-21 21:32:29.738 7 DEBUG nova.network.neutronv2.api [req-9efc8a32-d51d-491f-8b30-68251b826764 0744f0f5276043daa340ec785fc6d290 d19758cc71f946918f0864d1492f6b53 - default default] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Refreshing network info cache for port e6d2c677-28ce-46b9-be96-e8fa27757525 _get_instance_nw_info /usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py:1755
2021-03-21 21:32:29.769 7 ERROR nova.virt.libvirt.driver [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.redhat.local:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Forbidden: Input/output error: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.redhat.local:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Forbidden: Input/output error
2021-03-21 21:32:29.769 7 DEBUG nova.virt.libvirt.driver [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Migration operation thread notification thread_finished /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:9201
2021-03-21 21:32:30.074 7 DEBUG nova.virt.libvirt.migration [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] VM running on src, migration failed _log /usr/lib/python3.6/site-packages/nova/virt/libvirt/migration.py:419
2021-03-21 21:32:30.075 7 DEBUG nova.virt.libvirt.driver [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:9015
2021-03-21 21:32:30.075 7 ERROR nova.virt.libvirt.driver [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Migration operation has aborted




Version-Release number of selected component (if applicable):
RHOS-16.2-RHEL-8-20210310.n.1

How reproducible:
Apparently, 100%

Steps to Reproduce:
run any live migration test

Comment 1 Martin Schuppert 2021-03-23 12:00:21 UTC

this is a duplicate of BZ1926602 and fixed with puppet-tripleo-11.5.1-2.20210302224955.el8ost , the job is running libvirt version: 6.10.0 , but has puppet-tripleo-11.5.1-2.20210221004950.cb960b9.el8ost.1.noarch installed. As a result, the configured live_migration_uri has not the required &proxy=netcat parameter:

live_migration_uri=qemu+ssh://nova_migration@%s:2022/system?keyfile=/etc/nova/migration/identity

*** This bug has been marked as a duplicate of bug 1926602 ***