Bug 836135

Summary: spice migration: prevent race with libvirt
Product: Red Hat Enterprise Linux 6 Reporter: Yonit Halperin <yhalperi>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4CC: acathrow, bsarathy, dallan, dblechte, dyasny, dyuan, juzhang, mkenneth, mzhan, qzhang, rwu, shu, virt-maint, weizhan, ydu, zhpeng, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.10.2-2.el6 Doc Type: Bug Fix
Doc Text:
Cause: Spice needs some time at the end of migration to transfer internal state to destination. However, it needs helping hand from libvirt. Consequence: With previous code it was possible to kill source qemu (and spice server) before internal state was transmitted. This lead into unresponsive client on destination side. Fix: After qemu reports migration has finished, libvirt waits for spice server to migrate. Libvirt checks repeatedly on the monitor until the spice server reports success. Qemu is not terminated meanwhile. Result: Spice client won't get unresponsive anymore.
Story Points: ---
Clone Of: 836133 Environment:
Last Closed: 2013-02-21 07:18:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 836133    
Bug Blocks: 836123, 846910, 846911, 894020    
Attachments:
Description Flags
libvirtd.log none

Description Yonit Halperin 2012-06-28 07:48:52 UTC
+++ This bug was initially created as a clone of Bug #836133 +++

Description of problem:
In order to support seamless migration (bug 836123) Spice needs to migrate its state, from the migration src to the target, when migration is completed. However, when libvirt identifies that migration has completed, it closes the src qemu, preventing spice from completing its migration.
We want to add a QMP event for signaling libvirt when spice migration has completed. libvirt will need to wait for this event before closing the src.
(spice api already contains a migration_completion callback for qemu).

In order for spice to recognize whether the installed libvirt supports this qmp event, we will also need to add a "seamless-migration=on" parameter to spice arguments in qemu. Only if this parameter is set, spice will execute the seamless migration pathway.

Comment 2 Yonit Halperin 2012-07-24 09:05:06 UTC
Hi Michal,
In https://bugzilla.redhat.com/show_bug.cgi?id=836133#c6, Luiz raised a concern that a QMP event triggered by spice is not enough. What do you think? Are there any other requirements?

Thanks,
Yonit.

Comment 3 Dave Allan 2012-07-24 15:41:17 UTC
Yonit, Michal is on vacation this week, do you need an immediate answer?

Comment 4 Michal Privoznik 2012-07-30 08:11:26 UTC
Yonit,

I think we can extend the output of 'query-migrate' command by a boolean called 'spice-migration-complete', for instance. Because libvirt uses this command to poll on qemu like every 50ms. Like Luiz said, an event is a kind of tricky as libvirt might miss it. The scenario looks like this: with libvirt running an user starts an migration. Then he kills libvirtd and start it again after while. If qemu would send an event meanwhile it won't reach libvirt. However, when libvirt starts the second time, it knows there was a migration running and starts to poll again.

Comment 5 Yonit Halperin 2012-08-01 11:29:30 UTC
(In reply to comment #4)
> Yonit,
> 
> I think we can extend the output of 'query-migrate' command by a boolean
> called 'spice-migration-complete', for instance. Because libvirt uses this
ok, I'll send a patch to qemu. I hope it will be accepted.

> command to poll on qemu like every 50ms. Like Luiz said, an event is a kind
> of tricky as libvirt might miss it. The scenario looks like this: with
> libvirt running an user starts an migration. Then he kills libvirtd and
> start it again after while. If qemu would send an event meanwhile it won't
> reach libvirt. However, when libvirt starts the second time, it knows there
> was a migration running and starts to poll again.

Comment 6 Dave Allan 2012-08-29 15:50:49 UTC
Martin, can we get QA ack?

Comment 7 Min Zhan 2012-08-30 02:32:24 UTC
(In reply to comment #6)
> Martin, can we get QA ack?

QA ack is given. Remove needinfo flag.

Comment 8 Michal Privoznik 2012-09-13 17:06:36 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2012-September/msg00941.html

Comment 9 Michal Privoznik 2012-09-20 14:52:33 UTC
Yet another patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2012-September/msg01417.html

Comment 15 zhe peng 2012-10-10 08:25:00 UTC
Created attachment 624651 [details]
libvirtd.log

libvirtd.log

Comment 17 zhe peng 2012-10-10 08:55:07 UTC
Hi Michal:
    So quickly :), very thanks!

verify with build:
libvirt-0.10.2-2.el6.x86_64
qemu-kvm-0.12.1.2-2.320.el6.x86_64
spice-server-0.12.0-1.el6.x86_64
spice-gtk-0.14-2.el6.x86_64
spice-protocol-0.12.2-1.el6.noarch


step:
 1: prepare two machine for migrate 
 2: on source, create a windows guest with spice
 .....
  <graphics type='spice' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
 .....
 3: migrate guest to target
  #virsh migrate win7_64 qemu+ssh://target.redhat.com/system
 4: check qemu process,make sure seamless-migration=on
 #ps -ef | grep win7_64
qemu      8278     1 53 17:47 ?        00:26:51 /usr/libexec/qemu-kvm -name win7_64 -S -M rhel6.2.0 -enable-kvm -m 2048 -smp 4,sockets=4,cores=1,threads=1 -uuid d551e24f-1368-6ffb-02fd-01eb28301ab5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win7_64.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -drive file=/var/lib/libvirt/migrate/win7-64-virtio.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:38:7e:d1,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5906,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device qxl,id=video1,vram_size=67108864,bus=pci.0,addr=0x7 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 

 5: after migrate finished, check libvirt log
.....
61634 2012-10-10 21:40:18.071+0000: 1751: debug : qemuMonitorJSONIOProcessLine:166 : QEMU_MONITOR_RECV_REPLY: mon=0x7f53ac0cf700 reply={"return": {"status": "completed"}, "id": "libvirt-95"}
.....
61668 2012-10-10 21:40:18.071+0000: 1756: debug : virJSONValueToString:1133 : result={"execute":"query-spice","id":"libvirt-96"}
61669 2012-10-10 21:40:18.071+0000: 1756: debug : qemuMonitorJSONCommandWithFd:259 : Send command '{"execute":"query-spice","id":"libvirt-96"}' for write with FD -1
.....
61763 2012-10-10 21:40:18.073+0000: 1751: debug : qemuMonitorIOProcess:354 : QEMU_MONITOR_IO_PROCESS: mon=0x7f53ac0cf700 buf={"return": {"migrated": tr      ue, "enabled": true, "auth": "none", "port": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}^M
61764  len=134
61765 2012-10-10 21:40:18.073+0000: 1751: debug : qemuMonitorJSONIOProcessLine:146 : Line [{"return": {"migrated": true, "enabled": true, "auth": "none      ", "port": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}]
61766 2012-10-10 21:40:18.073+0000: 1751: debug : virJSONValueFromString:975 : string={"return": {"migrated": true, "enabled": true, "auth": "none", "p      ort": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}

verification passed.

Comment 18 Yonit Halperin 2012-10-18 06:30:21 UTC
Hi zhe,

Have you verified the bug fix with a remote viewer connection? Spice session migration is relevant only with a spice client session. According to the report in Bug 836133 comment #21, there is still a problem.

Comment 19 zhe peng 2012-10-18 07:10:24 UTC
Hi Yonit,

    No,when verify this bug, i dont use virt-viewer or remote-viewer to connect guest. I try it yesterday, use virt-viewer to connect guest then doing migrate,i always got error like "{"migrated": false," in libvirt log, but migrate can complete,  I will follow up bug: https://bugzilla.redhat.com/show_bug.cgi?id=836133 to track the issue.

Comment 22 errata-xmlrpc 2013-02-21 07:18:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html