Description of problem: Live Migration failure: operation failed: Failed to connect to remote libvirt URI Version-Release number of selected component (if applicable): tripleo-ansible-0.5.1-2.20201223225653.c876e30.el8ost.1.noarch openstack-nova-migration-20.4.2-2.20201224134938.81a3f4b.el8ost.1.noarch openstack-nova-compute-20.4.2-2.20201224134938.81a3f4b.el8ost.1.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP16.2 with 1 controller, 3 compute nodes, cinder with nfs storage. 2. Create VM from volume and check it's running on overcloud-novacompute-1 $ openstack server list|grep vol | 0404929c-b8db-4535-a4e0-bca31c3aaff1 | asb-vm-qcow2-vol | ACTIVE | asb-net1=192.168.33.135 | | | 3. Try to live migrate asb-vm-qcow2-vol from overcloud-novacompute-1 to overcloud-novacompute-2, the command line return without error (overcloud) [stack@dell-per730-44 ~]$ openstack server migrate --live-migration asb-vm-qcow2-vol (overcloud) [stack@dell-per730-44 ~]$ echo $? 0 4. Check the log in nova-compute.log, there is "error: Failed to connect to remote libvirt URI" [heat-admin@overcloud-novacompute-1 ~]$ sudo tail -f /var/log/containers/nova/nova-compute.log 2021-02-09 07:46:41.806 7 INFO nova.compute.manager [-] [instance: 0404929c-b8db-4535-a4e0-bca31c3aaff1] Took 3.26 seconds for pre_live_migration on destination host overcloud-novacompute-2.localdomain. 2021-02-09 07:46:42.488 7 ERROR nova.virt.libvirt.driver [-] [instance: 0404929c-b8db-4535-a4e0-bca31c3aaff1] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.localdomain:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Forbidden: Input/output error: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.localdomain:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Forbidden: Input/output error 2021-02-09 07:46:42.716 7 ERROR nova.virt.libvirt.driver [-] [instance: 0404929c-b8db-4535-a4e0-bca31c3aaff1] Migration operation has aborted 2021-02-09 07:46:42.737 7 INFO nova.compute.manager [-] [instance: 0404929c-b8db-4535-a4e0-bca31c3aaff1] Swapping old allocation on dict_keys(['117f60e4-6c4b-4ed6-88c3-5b7add7d197c']) held by migration cfd0a7dd-0e7a-47cb-ab30-e322ecd42d23 for instance Actual results: In step3: the command line return without error In step4: the live migration is failed Expected results: In step3: the command line return error, if the live migration is failed In step4: the live migration should success and no error in the nova-compute.log Additional info:
Created attachment 1755862 [details] nova-compute.log
Can we have full sosreports from the compute node please, or access to the node?
Created attachment 1758774 [details] source_novacompute-1_log.tgz
Created attachment 1758775 [details] target_novacompute-2_log.tgz
Hi, Jiri This bug blocked the live migration on OSP16.2, I attached the libvirtd logs in files: source_novacompute-1_log.tgz, target_novacompute-2_log.tgz, will you please help to confirm if it need libvirt code change? Many thanks! Please notice: In order to collect the libvirtd debug log, after reproduce this issue, I systemctl restart tripleo_nova_libvirt.service, then reran the reproduce steps. Regards, Chenli Hu
This is most likely a configuration issue on the destination host. Check that the nova_migration user is allowed to properly connect to libvirtd on that hosts. If not, check authorization or access control method enabled in libvirtd.conf and their configuration.
Thanks Jiri! Normally we don't need to do any configuration for live migration in OSP after deployment, then the code changed should be in OSP, let's waiting for their debugging ...
Created attachment 1760815 [details] containers-prepare-parameter-prepare.yaml and Dockfiles
*** Bug 1942000 has been marked as a duplicate of this bug. ***
*** Bug 1942881 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483