Description of problem: Several tempest tests failed during VM live migration (cold migration tests pass): https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv4-geneve/101/testReport/ For example, the test test_server_connectivity_live_migration fails because migration is not performed: AssertionError: 'compute-0.redhat.local' == 'compute-0.redhat.local' : Server did not migrate compute-0's nova-compute logs show the following error: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv4-geneve/101/compute-0/var/log/containers/nova/nova-compute.log.1.gz 2021-03-21 21:32:29.738 7 DEBUG nova.network.neutronv2.api [req-9efc8a32-d51d-491f-8b30-68251b826764 0744f0f5276043daa340ec785fc6d290 d19758cc71f946918f0864d1492f6b53 - default default] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Refreshing network info cache for port e6d2c677-28ce-46b9-be96-e8fa27757525 _get_instance_nw_info /usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py:1755 2021-03-21 21:32:29.769 7 ERROR nova.virt.libvirt.driver [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.redhat.local:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Forbidden: Input/output error: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.redhat.local:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Forbidden: Input/output error 2021-03-21 21:32:29.769 7 DEBUG nova.virt.libvirt.driver [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Migration operation thread notification thread_finished /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:9201 2021-03-21 21:32:30.074 7 DEBUG nova.virt.libvirt.migration [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] VM running on src, migration failed _log /usr/lib/python3.6/site-packages/nova/virt/libvirt/migration.py:419 2021-03-21 21:32:30.075 7 DEBUG nova.virt.libvirt.driver [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:9015 2021-03-21 21:32:30.075 7 ERROR nova.virt.libvirt.driver [-] [instance: ea6b9516-b413-4bdb-bdaf-63949e6e93e0] Migration operation has aborted Version-Release number of selected component (if applicable): RHOS-16.2-RHEL-8-20210310.n.1 How reproducible: Apparently, 100% Steps to Reproduce: run any live migration test
this is a duplicate of BZ1926602 and fixed with puppet-tripleo-11.5.1-2.20210302224955.el8ost , the job is running libvirt version: 6.10.0 , but has puppet-tripleo-11.5.1-2.20210221004950.cb960b9.el8ost.1.noarch installed. As a result, the configured live_migration_uri has not the required &proxy=netcat parameter: live_migration_uri=qemu+ssh://nova_migration@%s:2022/system?keyfile=/etc/nova/migration/identity *** This bug has been marked as a duplicate of bug 1926602 ***