Bug 836135 - spice migration: prevent race with libvirt
spice migration: prevent race with libvirt
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Michal Privoznik
Virtualization Bugs
:
Depends On: 836133
Blocks: 836123 846910 846911 894020
  Show dependency treegraph
 
Reported: 2012-06-28 03:48 EDT by Yonit Halperin
Modified: 2013-02-21 02:18 EST (History)
17 users (show)

See Also:
Fixed In Version: libvirt-0.10.2-2.el6
Doc Type: Bug Fix
Doc Text:
Cause: Spice needs some time at the end of migration to transfer internal state to destination. However, it needs helping hand from libvirt. Consequence: With previous code it was possible to kill source qemu (and spice server) before internal state was transmitted. This lead into unresponsive client on destination side. Fix: After qemu reports migration has finished, libvirt waits for spice server to migrate. Libvirt checks repeatedly on the monitor until the spice server reports success. Qemu is not terminated meanwhile. Result: Spice client won't get unresponsive anymore.
Story Points: ---
Clone Of: 836133
Environment:
Last Closed: 2013-02-21 02:18:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
libvirtd.log (8.24 MB, text/plain)
2012-10-10 04:25 EDT, zhe peng
no flags Details

  None (edit)
Description Yonit Halperin 2012-06-28 03:48:52 EDT
+++ This bug was initially created as a clone of Bug #836133 +++

Description of problem:
In order to support seamless migration (bug 836123) Spice needs to migrate its state, from the migration src to the target, when migration is completed. However, when libvirt identifies that migration has completed, it closes the src qemu, preventing spice from completing its migration.
We want to add a QMP event for signaling libvirt when spice migration has completed. libvirt will need to wait for this event before closing the src.
(spice api already contains a migration_completion callback for qemu).

In order for spice to recognize whether the installed libvirt supports this qmp event, we will also need to add a "seamless-migration=on" parameter to spice arguments in qemu. Only if this parameter is set, spice will execute the seamless migration pathway.
Comment 2 Yonit Halperin 2012-07-24 05:05:06 EDT
Hi Michal,
In https://bugzilla.redhat.com/show_bug.cgi?id=836133#c6, Luiz raised a concern that a QMP event triggered by spice is not enough. What do you think? Are there any other requirements?

Thanks,
Yonit.
Comment 3 Dave Allan 2012-07-24 11:41:17 EDT
Yonit, Michal is on vacation this week, do you need an immediate answer?
Comment 4 Michal Privoznik 2012-07-30 04:11:26 EDT
Yonit,

I think we can extend the output of 'query-migrate' command by a boolean called 'spice-migration-complete', for instance. Because libvirt uses this command to poll on qemu like every 50ms. Like Luiz said, an event is a kind of tricky as libvirt might miss it. The scenario looks like this: with libvirt running an user starts an migration. Then he kills libvirtd and start it again after while. If qemu would send an event meanwhile it won't reach libvirt. However, when libvirt starts the second time, it knows there was a migration running and starts to poll again.
Comment 5 Yonit Halperin 2012-08-01 07:29:30 EDT
(In reply to comment #4)
> Yonit,
> 
> I think we can extend the output of 'query-migrate' command by a boolean
> called 'spice-migration-complete', for instance. Because libvirt uses this
ok, I'll send a patch to qemu. I hope it will be accepted.

> command to poll on qemu like every 50ms. Like Luiz said, an event is a kind
> of tricky as libvirt might miss it. The scenario looks like this: with
> libvirt running an user starts an migration. Then he kills libvirtd and
> start it again after while. If qemu would send an event meanwhile it won't
> reach libvirt. However, when libvirt starts the second time, it knows there
> was a migration running and starts to poll again.
Comment 6 Dave Allan 2012-08-29 11:50:49 EDT
Martin, can we get QA ack?
Comment 7 min zhan 2012-08-29 22:32:24 EDT
(In reply to comment #6)
> Martin, can we get QA ack?

QA ack is given. Remove needinfo flag.
Comment 8 Michal Privoznik 2012-09-13 13:06:36 EDT
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2012-September/msg00941.html
Comment 9 Michal Privoznik 2012-09-20 10:52:33 EDT
Yet another patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2012-September/msg01417.html
Comment 15 zhe peng 2012-10-10 04:25:00 EDT
Created attachment 624651 [details]
libvirtd.log

libvirtd.log
Comment 17 zhe peng 2012-10-10 04:55:07 EDT
Hi Michal:
    So quickly :), very thanks!

verify with build:
libvirt-0.10.2-2.el6.x86_64
qemu-kvm-0.12.1.2-2.320.el6.x86_64
spice-server-0.12.0-1.el6.x86_64
spice-gtk-0.14-2.el6.x86_64
spice-protocol-0.12.2-1.el6.noarch


step:
 1: prepare two machine for migrate 
 2: on source, create a windows guest with spice
 .....
  <graphics type='spice' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
 .....
 3: migrate guest to target
  #virsh migrate win7_64 qemu+ssh://target.redhat.com/system
 4: check qemu process,make sure seamless-migration=on
 #ps -ef | grep win7_64
qemu      8278     1 53 17:47 ?        00:26:51 /usr/libexec/qemu-kvm -name win7_64 -S -M rhel6.2.0 -enable-kvm -m 2048 -smp 4,sockets=4,cores=1,threads=1 -uuid d551e24f-1368-6ffb-02fd-01eb28301ab5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win7_64.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -drive file=/var/lib/libvirt/migrate/win7-64-virtio.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:38:7e:d1,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5906,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device qxl,id=video1,vram_size=67108864,bus=pci.0,addr=0x7 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 

 5: after migrate finished, check libvirt log
.....
61634 2012-10-10 21:40:18.071+0000: 1751: debug : qemuMonitorJSONIOProcessLine:166 : QEMU_MONITOR_RECV_REPLY: mon=0x7f53ac0cf700 reply={"return": {"status": "completed"}, "id": "libvirt-95"}
.....
61668 2012-10-10 21:40:18.071+0000: 1756: debug : virJSONValueToString:1133 : result={"execute":"query-spice","id":"libvirt-96"}
61669 2012-10-10 21:40:18.071+0000: 1756: debug : qemuMonitorJSONCommandWithFd:259 : Send command '{"execute":"query-spice","id":"libvirt-96"}' for write with FD -1
.....
61763 2012-10-10 21:40:18.073+0000: 1751: debug : qemuMonitorIOProcess:354 : QEMU_MONITOR_IO_PROCESS: mon=0x7f53ac0cf700 buf={"return": {"migrated": tr      ue, "enabled": true, "auth": "none", "port": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}^M
61764  len=134
61765 2012-10-10 21:40:18.073+0000: 1751: debug : qemuMonitorJSONIOProcessLine:146 : Line [{"return": {"migrated": true, "enabled": true, "auth": "none      ", "port": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}]
61766 2012-10-10 21:40:18.073+0000: 1751: debug : virJSONValueFromString:975 : string={"return": {"migrated": true, "enabled": true, "auth": "none", "p      ort": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}

verification passed.
Comment 18 Yonit Halperin 2012-10-18 02:30:21 EDT
Hi zhe,

Have you verified the bug fix with a remote viewer connection? Spice session migration is relevant only with a spice client session. According to the report in Bug 836133 comment #21, there is still a problem.
Comment 19 zhe peng 2012-10-18 03:10:24 EDT
Hi Yonit,

    No,when verify this bug, i dont use virt-viewer or remote-viewer to connect guest. I try it yesterday, use virt-viewer to connect guest then doing migrate,i always got error like "{"migrated": false," in libvirt log, but migrate can complete,  I will follow up bug: https://bugzilla.redhat.com/show_bug.cgi?id=836133 to track the issue.
Comment 22 errata-xmlrpc 2013-02-21 02:18:11 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html

Note You need to log in before you can comment on or make changes to this bug.