Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1339910 - Migration with "--xml <new_xml_file>" stuck in confirm phase when a spice client(via virt-manager) is connected.
Summary: Migration with "--xml <new_xml_file>" stuck in confirm phase when a spice cli...
Keywords:
Status: CLOSED DUPLICATE of bug 1352836
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-26 06:57 UTC by Fangge Jin
Modified: 2016-09-14 10:45 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-18 14:55:16 UTC
Target Upstream Version:


Attachments (Terms of Use)
Guest XML and libvirtd/qemu logs (1.80 MB, application/x-gzip)
2016-05-26 06:57 UTC, Fangge Jin
no flags Details

Description Fangge Jin 2016-05-26 06:57:32 UTC
Created attachment 1161750 [details]
Guest XML and libvirtd/qemu logs

Description of problem:
Connect to guest spice graphic by virt-manager, then migrate guest to another host with --xml <new_xml> (the new xml file provides spice password), migration will be stuck in confirm phase.

Version-Release number of selected component:
libvirt-1.3.4-1.el7.x86_64
qemu-kvm-rhev-2.6.0-2.el7.x86_64
spice-server-0.12.4-17.el7.x86_64
virt-manager-1.3.2-1.el7.noarch
spice-gtk3-0.31-2.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1. On source host:
0) Prepare a guest with spice graphic:
    <graphics type='spice' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>

1)# virsh start rhel7.2
Domain rhel7.2 started

2) Launch virt-manager, open graphic of guest rhel7.2

3)Dump guest xml to a file and add password setting for spice graphic
# virsh dumpxml rhel7.2 > /tmp/rhel7.2.xml.passwd
# vi /tmp/rhel7.2.xml.passwd
...
    <graphics type='spice' **passwd='redhat'** port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
...

4) Do migration:
# virsh migrate rhel7.2 qemu+ssh://hp-dl385g7-06.lab.eng.pek2.redhat.com/system --verbose --live --xml /tmp/rhel7.2.xml.passwd
Migration: [100 %]

The virsh command doesn't exit (migration stuck in confirm phase) after the memory transfer is 100% completed

The backstace of libvirtd:
Thread 15 (Thread 0x7f1a6ca6e700 (LWP 19597)):
#0  0x00007f1a7a4e76d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f1a7ce80bd6 in virCondWait (c=c@entry=0x7f1a1c200df8, m=m@entry=0x7f1a1c200dd0) at util/virthread.c:154
#2  0x00007f1a7ce9fd05 in virDomainObjWait (vm=vm@entry=0x7f1a1c200dc0) at conf/domain_conf.c:2825
#3  0x00007f1a601f5587 in qemuMigrationWaitForSpice (vm=0x7f1a1c200dc0) at qemu/qemu_migration.c:2512
#4  qemuMigrationConfirmPhase (driver=driver@entry=0x7f1a1c0edb50, conn=conn@entry=0x7f1a3c000c50, vm=0x7f1a1c200dc0, cookiein=cookiein@entry=0x7f1a5c0020a0 "<qemu-migration>\n  <name>rhel7.2</name>\n  <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n  <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n  <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=cookieinlen@entry=1072, flags=flags@entry=257, retcode=retcode@entry=0) at qemu/qemu_migration.c:4116
#5  0x00007f1a601f7eec in qemuMigrationConfirm (conn=0x7f1a3c000c50, vm=0x7f1a1c200dc0, cookiein=cookiein@entry=0x7f1a5c0020a0 "<qemu-migration>\n  <name>rhel7.2</name>\n  <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n  <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n  <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=cookieinlen@entry=1072, flags=flags@entry=257, cancelled=cancelled@entry=0) at qemu/qemu_migration.c:4189
#6  0x00007f1a60225732 in qemuDomainMigrateConfirm3Params (domain=0x7f1a5c002550, params=<optimized out>, nparams=<optimized out>, cookiein=0x7f1a5c0020a0 "<qemu-migration>\n  <name>rhel7.2</name>\n  <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n  <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n  <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=1072, flags=257, cancelled=0) at qemu/qemu_driver.c:12838
#7  0x00007f1a7cf23587 in virDomainMigrateConfirm3Params (domain=domain@entry=0x7f1a5c002550, params=0x7f1a5c002690, nparams=3, cookiein=0x7f1a5c0020a0 "<qemu-migration>\n  <name>rhel7.2</name>\n  <uuid>895c72bc-19f1-4f16-84da-d3e329d807ef</uuid>\n  <hostname>hp-dl385g7-06.lab.eng.pek2.redhat.com</hostname>\n  <hostuuid>30333735-3938-4e43-4731-34325354425"..., cookieinlen=1072, flags=257, cancelled=0) at libvirt-domain.c:5329
#8  0x00007f1a7db675d6 in remoteDispatchDomainMigrateConfirm3Params (server=0x7f1a7fd0efb0, msg=0x7f1a7fd28ad0, args=0x7f1a5c000e30, rerr=0x7f1a6ca6dc10, client=<optimized out>) at remote.c:5139
#9  remoteDispatchDomainMigrateConfirm3ParamsHelper (server=0x7f1a7fd0efb0, client=<optimized out>, msg=0x7f1a7fd28ad0, rerr=0x7f1a6ca6dc10, args=0x7f1a5c000e30, ret=0x7f1a5c000bf0) at remote_dispatch.h:7124
#10 0x00007f1a7cf91832 in virNetServerProgramDispatchCall (msg=0x7f1a7fd28ad0, client=0x7f1a7fd271e0, server=0x7f1a7fd0efb0, prog=0x7f1a7fd224d0) at rpc/virnetserverprogram.c:437
#11 virNetServerProgramDispatch (prog=0x7f1a7fd224d0, server=server@entry=0x7f1a7fd0efb0, client=0x7f1a7fd271e0, msg=0x7f1a7fd28ad0) at rpc/virnetserverprogram.c:307
#12 0x00007f1a7db9540d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f1a7fd0efb0) at rpc/virnetserver.c:137
#13 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f1a7fd0efb0) at rpc/virnetserver.c:158
#14 0x00007f1a7ce815e1 in virThreadPoolWorker (opaque=opaque@entry=0x7f1a7fd0e9d0) at util/virthreadpool.c:167
#15 0x00007f1a7ce80968 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#16 0x00007f1a7a4e3dc5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f1a7a2111cd in clone () from /lib64/libc.so.6


2. On target host:
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     rhel7.2                        running

3. On source host:
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     rhel7.2                        paused

#virsh destroy rhel7.2

# ll /90121/fjin/rhel7.2.qcow2
-rw-------. 1 root root 9665380352 5月  25 2016 /90121/fjin/rhel7.2.qcow2
              ^^^^ ^^^^
The ownership of guest image is changed to "root" by source libvirtd. So the guest on target host can't access the disk.


Actual results:
Migration stuck in confirm phase on source host.
And destroy guest on source host will change the ownership of guest image.

Expected results:
Migration can finish completely.
Destroy guest on source host doesn't change the ownership of guest image.

Additional info:
1) Sometimes the souce libvirtd backstrace is as below:
Thread 17 (Thread 0x7f53331c1700 (LWP 18869)):
#0  0x00007f533fc386d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f53425d1bd6 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f53425f0d05 in virDomainObjWait () from /lib64/libvirt.so.0
#3  0x00007f5321814587 in qemuMigrationConfirmPhase () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f5321816eec in qemuMigrationConfirm () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f5321844732 in qemuDomainMigrateConfirm3Params () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f5342674587 in virDomainMigrateConfirm3Params () from /lib64/libvirt.so.0
#7  0x00007f53432b85d6 in remoteDispatchDomainMigrateConfirm3ParamsHelper ()
#8  0x00007f53426e2832 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#9  0x00007f53432e640d in virNetServerHandleJob ()
#10 0x00007f53425d25e1 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f53425d1968 in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f533fc34dc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f533f9621cd in clone () from /lib64/libc.so.6

2) Migration can finish completely with remote-viewer

Comment 2 Fangge Jin 2016-06-01 09:17:14 UTC
The following two scenarios can also reproduce this bug:
1.Set different spice_password value (in qemu.conf) on source and target host
2.Disable spice_password on source host, while enable it on target host


This scenario can't reproduce this bug:
1.Enable spice_password on source host, while disable it on target host

Comment 3 Fangge Jin 2016-06-01 09:38:50 UTC
(In reply to JinFangge from comment #2)
> The following two scenarios can also reproduce this bug:
> 1.Set different spice_password value (in qemu.conf) on source and target host
> 2.Disable spice_password on source host, while enable it on target host
> 
> 
> This scenario can't reproduce this bug:
> 1.Enable spice_password on source host, while disable it on target host

And these scenarios can be reproduced with both remote-viewer and virt-manager

Comment 4 Jiri Denemark 2016-07-18 14:55:16 UTC
This is just another instance of bug 1352836.

*** This bug has been marked as a duplicate of bug 1352836 ***

Comment 5 Fangge Jin 2016-09-14 10:45:12 UTC
Test again with the following builds:
libvirt-2.0.0-8.el7.x86_64
qemu-kvm-rhev-2.6.0-25.el7.x86_64
spice-server-0.12.4-19.el7.x86_64

Steps are exactly same as the bug description, spice migration can finish and virsh command exits after migration finished.


Note You need to log in before you can comment on or make changes to this bug.