Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1161750[details]
Guest XML and libvirtd/qemu logs
Description of problem:
Connect to guest spice graphic by virt-manager, then migrate guest to another host with --xml <new_xml> (the new xml file provides spice password), migration will be stuck in confirm phase.
Version-Release number of selected component:
libvirt-1.3.4-1.el7.x86_64
qemu-kvm-rhev-2.6.0-2.el7.x86_64
spice-server-0.12.4-17.el7.x86_64
virt-manager-1.3.2-1.el7.noarch
spice-gtk3-0.31-2.el7.x86_64
How reproducible:
100%
Steps to Reproduce:
1. On source host:
0) Prepare a guest with spice graphic:
<graphics type='spice' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
1)# virsh start rhel7.2
Domain rhel7.2 started
2) Launch virt-manager, open graphic of guest rhel7.2
3)Dump guest xml to a file and add password setting for spice graphic
# virsh dumpxml rhel7.2 > /tmp/rhel7.2.xml.passwd
# vi /tmp/rhel7.2.xml.passwd
...
<graphics type='spice' **passwd='redhat'** port='5900' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
...
4) Do migration:
# virsh migrate rhel7.2 qemu+ssh://hp-dl385g7-06.lab.eng.pek2.redhat.com/system --verbose --live --xml /tmp/rhel7.2.xml.passwd
Migration: [100 %]
The virsh command doesn't exit (migration stuck in confirm phase) after the memory transfer is 100% completed
The backstace of libvirtd:
Thread 15 (Thread 0x7f1a6ca6e700 (LWP 19597)):
#0 0x00007f1a7a4e76d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a7ce80bd6 in virCondWait (c=c@entry=0x7f1a1c200df8, m=m@entry=0x7f1a1c200dd0) at util/virthread.c:154
#2 0x00007f1a7ce9fd05 in virDomainObjWait (vm=vm@entry=0x7f1a1c200dc0) at conf/domain_conf.c:2825
#3 0x00007f1a601f5587 in qemuMigrationWaitForSpice (vm=0x7f1a1c200dc0) at qemu/qemu_migration.c:2512
#4 qemuMigrationConfirmPhase (driver=driver@entry=0x7f1a1c0edb50, conn=conn@entry=0x7f1a3c000c50, vm=0x7f1a1c200dc0, cookiein=cookiein@entry=0x7f1a5c0020a0 "<qemu-migration>\n <name>rhel7.2</name>\n <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=cookieinlen@entry=1072, flags=flags@entry=257, retcode=retcode@entry=0) at qemu/qemu_migration.c:4116
#5 0x00007f1a601f7eec in qemuMigrationConfirm (conn=0x7f1a3c000c50, vm=0x7f1a1c200dc0, cookiein=cookiein@entry=0x7f1a5c0020a0 "<qemu-migration>\n <name>rhel7.2</name>\n <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=cookieinlen@entry=1072, flags=flags@entry=257, cancelled=cancelled@entry=0) at qemu/qemu_migration.c:4189
#6 0x00007f1a60225732 in qemuDomainMigrateConfirm3Params (domain=0x7f1a5c002550, params=<optimized out>, nparams=<optimized out>, cookiein=0x7f1a5c0020a0 "<qemu-migration>\n <name>rhel7.2</name>\n <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=1072, flags=257, cancelled=0) at qemu/qemu_driver.c:12838
#7 0x00007f1a7cf23587 in virDomainMigrateConfirm3Params (domain=domain@entry=0x7f1a5c002550, params=0x7f1a5c002690, nparams=3, cookiein=0x7f1a5c0020a0 "<qemu-migration>\n <name>rhel7.2</name>\n <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=1072, flags=257, cancelled=0) at libvirt-domain.c:5329
#8 0x00007f1a7db675d6 in remoteDispatchDomainMigrateConfirm3Params (server=0x7f1a7fd0efb0, msg=0x7f1a7fd28ad0, args=0x7f1a5c000e30, rerr=0x7f1a6ca6dc10, client=<optimized out>) at remote.c:5139
#9 remoteDispatchDomainMigrateConfirm3ParamsHelper (server=0x7f1a7fd0efb0, client=<optimized out>, msg=0x7f1a7fd28ad0, rerr=0x7f1a6ca6dc10, args=0x7f1a5c000e30, ret=0x7f1a5c000bf0) at remote_dispatch.h:7124
#10 0x00007f1a7cf91832 in virNetServerProgramDispatchCall (msg=0x7f1a7fd28ad0, client=0x7f1a7fd271e0, server=0x7f1a7fd0efb0, prog=0x7f1a7fd224d0) at rpc/virnetserverprogram.c:437
#11 virNetServerProgramDispatch (prog=0x7f1a7fd224d0, server=server@entry=0x7f1a7fd0efb0, client=0x7f1a7fd271e0, msg=0x7f1a7fd28ad0) at rpc/virnetserverprogram.c:307
#12 0x00007f1a7db9540d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f1a7fd0efb0) at rpc/virnetserver.c:137
#13 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f1a7fd0efb0) at rpc/virnetserver.c:158
#14 0x00007f1a7ce815e1 in virThreadPoolWorker (opaque=opaque@entry=0x7f1a7fd0e9d0) at util/virthreadpool.c:167
#15 0x00007f1a7ce80968 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#16 0x00007f1a7a4e3dc5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f1a7a2111cd in clone () from /lib64/libc.so.6
2. On target host:
# virsh list
Id Name State
----------------------------------------------------
1 rhel7.2 running
3. On source host:
# virsh list
Id Name State
----------------------------------------------------
1 rhel7.2 paused
#virsh destroy rhel7.2
# ll /90121/fjin/rhel7.2.qcow2
-rw-------. 1 root root 9665380352 5月 25 2016 /90121/fjin/rhel7.2.qcow2
^^^^ ^^^^
The ownership of guest image is changed to "root" by source libvirtd. So the guest on target host can't access the disk.
Actual results:
Migration stuck in confirm phase on source host.
And destroy guest on source host will change the ownership of guest image.
Expected results:
Migration can finish completely.
Destroy guest on source host doesn't change the ownership of guest image.
Additional info:
1) Sometimes the souce libvirtd backstrace is as below:
Thread 17 (Thread 0x7f53331c1700 (LWP 18869)):
#0 0x00007f533fc386d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f53425d1bd6 in virCondWait () from /lib64/libvirt.so.0
#2 0x00007f53425f0d05 in virDomainObjWait () from /lib64/libvirt.so.0
#3 0x00007f5321814587 in qemuMigrationConfirmPhase () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f5321816eec in qemuMigrationConfirm () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f5321844732 in qemuDomainMigrateConfirm3Params () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f5342674587 in virDomainMigrateConfirm3Params () from /lib64/libvirt.so.0
#7 0x00007f53432b85d6 in remoteDispatchDomainMigrateConfirm3ParamsHelper ()
#8 0x00007f53426e2832 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#9 0x00007f53432e640d in virNetServerHandleJob ()
#10 0x00007f53425d25e1 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f53425d1968 in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f533fc34dc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f533f9621cd in clone () from /lib64/libc.so.6
2) Migration can finish completely with remote-viewer
The following two scenarios can also reproduce this bug:
1.Set different spice_password value (in qemu.conf) on source and target host
2.Disable spice_password on source host, while enable it on target host
This scenario can't reproduce this bug:
1.Enable spice_password on source host, while disable it on target host
(In reply to JinFangge from comment #2)
> The following two scenarios can also reproduce this bug:
> 1.Set different spice_password value (in qemu.conf) on source and target host
> 2.Disable spice_password on source host, while enable it on target host
>
>
> This scenario can't reproduce this bug:
> 1.Enable spice_password on source host, while disable it on target host
And these scenarios can be reproduced with both remote-viewer and virt-manager
Test again with the following builds:
libvirt-2.0.0-8.el7.x86_64
qemu-kvm-rhev-2.6.0-25.el7.x86_64
spice-server-0.12.4-19.el7.x86_64
Steps are exactly same as the bug description, spice migration can finish and virsh command exits after migration finished.
Created attachment 1161750 [details] Guest XML and libvirtd/qemu logs Description of problem: Connect to guest spice graphic by virt-manager, then migrate guest to another host with --xml <new_xml> (the new xml file provides spice password), migration will be stuck in confirm phase. Version-Release number of selected component: libvirt-1.3.4-1.el7.x86_64 qemu-kvm-rhev-2.6.0-2.el7.x86_64 spice-server-0.12.4-17.el7.x86_64 virt-manager-1.3.2-1.el7.noarch spice-gtk3-0.31-2.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. On source host: 0) Prepare a guest with spice graphic: <graphics type='spice' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> 1)# virsh start rhel7.2 Domain rhel7.2 started 2) Launch virt-manager, open graphic of guest rhel7.2 3)Dump guest xml to a file and add password setting for spice graphic # virsh dumpxml rhel7.2 > /tmp/rhel7.2.xml.passwd # vi /tmp/rhel7.2.xml.passwd ... <graphics type='spice' **passwd='redhat'** port='5900' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> ... 4) Do migration: # virsh migrate rhel7.2 qemu+ssh://hp-dl385g7-06.lab.eng.pek2.redhat.com/system --verbose --live --xml /tmp/rhel7.2.xml.passwd Migration: [100 %] The virsh command doesn't exit (migration stuck in confirm phase) after the memory transfer is 100% completed The backstace of libvirtd: Thread 15 (Thread 0x7f1a6ca6e700 (LWP 19597)): #0 0x00007f1a7a4e76d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1a7ce80bd6 in virCondWait (c=c@entry=0x7f1a1c200df8, m=m@entry=0x7f1a1c200dd0) at util/virthread.c:154 #2 0x00007f1a7ce9fd05 in virDomainObjWait (vm=vm@entry=0x7f1a1c200dc0) at conf/domain_conf.c:2825 #3 0x00007f1a601f5587 in qemuMigrationWaitForSpice (vm=0x7f1a1c200dc0) at qemu/qemu_migration.c:2512 #4 qemuMigrationConfirmPhase (driver=driver@entry=0x7f1a1c0edb50, conn=conn@entry=0x7f1a3c000c50, vm=0x7f1a1c200dc0, cookiein=cookiein@entry=0x7f1a5c0020a0 "<qemu-migration>\n <name>rhel7.2</name>\n <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=cookieinlen@entry=1072, flags=flags@entry=257, retcode=retcode@entry=0) at qemu/qemu_migration.c:4116 #5 0x00007f1a601f7eec in qemuMigrationConfirm (conn=0x7f1a3c000c50, vm=0x7f1a1c200dc0, cookiein=cookiein@entry=0x7f1a5c0020a0 "<qemu-migration>\n <name>rhel7.2</name>\n <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=cookieinlen@entry=1072, flags=flags@entry=257, cancelled=cancelled@entry=0) at qemu/qemu_migration.c:4189 #6 0x00007f1a60225732 in qemuDomainMigrateConfirm3Params (domain=0x7f1a5c002550, params=<optimized out>, nparams=<optimized out>, cookiein=0x7f1a5c0020a0 "<qemu-migration>\n <name>rhel7.2</name>\n <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=1072, flags=257, cancelled=0) at qemu/qemu_driver.c:12838 #7 0x00007f1a7cf23587 in virDomainMigrateConfirm3Params (domain=domain@entry=0x7f1a5c002550, params=0x7f1a5c002690, nparams=3, cookiein=0x7f1a5c0020a0 "<qemu-migration>\n <name>rhel7.2</name>\n <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=1072, flags=257, cancelled=0) at libvirt-domain.c:5329 #8 0x00007f1a7db675d6 in remoteDispatchDomainMigrateConfirm3Params (server=0x7f1a7fd0efb0, msg=0x7f1a7fd28ad0, args=0x7f1a5c000e30, rerr=0x7f1a6ca6dc10, client=<optimized out>) at remote.c:5139 #9 remoteDispatchDomainMigrateConfirm3ParamsHelper (server=0x7f1a7fd0efb0, client=<optimized out>, msg=0x7f1a7fd28ad0, rerr=0x7f1a6ca6dc10, args=0x7f1a5c000e30, ret=0x7f1a5c000bf0) at remote_dispatch.h:7124 #10 0x00007f1a7cf91832 in virNetServerProgramDispatchCall (msg=0x7f1a7fd28ad0, client=0x7f1a7fd271e0, server=0x7f1a7fd0efb0, prog=0x7f1a7fd224d0) at rpc/virnetserverprogram.c:437 #11 virNetServerProgramDispatch (prog=0x7f1a7fd224d0, server=server@entry=0x7f1a7fd0efb0, client=0x7f1a7fd271e0, msg=0x7f1a7fd28ad0) at rpc/virnetserverprogram.c:307 #12 0x00007f1a7db9540d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f1a7fd0efb0) at rpc/virnetserver.c:137 #13 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f1a7fd0efb0) at rpc/virnetserver.c:158 #14 0x00007f1a7ce815e1 in virThreadPoolWorker (opaque=opaque@entry=0x7f1a7fd0e9d0) at util/virthreadpool.c:167 #15 0x00007f1a7ce80968 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #16 0x00007f1a7a4e3dc5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f1a7a2111cd in clone () from /lib64/libc.so.6 2. On target host: # virsh list Id Name State ---------------------------------------------------- 1 rhel7.2 running 3. On source host: # virsh list Id Name State ---------------------------------------------------- 1 rhel7.2 paused #virsh destroy rhel7.2 # ll /90121/fjin/rhel7.2.qcow2 -rw-------. 1 root root 9665380352 5月 25 2016 /90121/fjin/rhel7.2.qcow2 ^^^^ ^^^^ The ownership of guest image is changed to "root" by source libvirtd. So the guest on target host can't access the disk. Actual results: Migration stuck in confirm phase on source host. And destroy guest on source host will change the ownership of guest image. Expected results: Migration can finish completely. Destroy guest on source host doesn't change the ownership of guest image. Additional info: 1) Sometimes the souce libvirtd backstrace is as below: Thread 17 (Thread 0x7f53331c1700 (LWP 18869)): #0 0x00007f533fc386d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53425d1bd6 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f53425f0d05 in virDomainObjWait () from /lib64/libvirt.so.0 #3 0x00007f5321814587 in qemuMigrationConfirmPhase () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f5321816eec in qemuMigrationConfirm () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f5321844732 in qemuDomainMigrateConfirm3Params () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f5342674587 in virDomainMigrateConfirm3Params () from /lib64/libvirt.so.0 #7 0x00007f53432b85d6 in remoteDispatchDomainMigrateConfirm3ParamsHelper () #8 0x00007f53426e2832 in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #9 0x00007f53432e640d in virNetServerHandleJob () #10 0x00007f53425d25e1 in virThreadPoolWorker () from /lib64/libvirt.so.0 #11 0x00007f53425d1968 in virThreadHelper () from /lib64/libvirt.so.0 #12 0x00007f533fc34dc5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f533f9621cd in clone () from /lib64/libc.so.6 2) Migration can finish completely with remote-viewer