RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 836135 - spice migration: prevent race with libvirt
Summary: spice migration: prevent race with libvirt
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 836133
Blocks: 836123 846910 846911 894020
TreeView+ depends on / blocked
 
Reported: 2012-06-28 07:48 UTC by Yonit Halperin
Modified: 2013-02-21 07:18 UTC (History)
17 users (show)

Fixed In Version: libvirt-0.10.2-2.el6
Doc Type: Bug Fix
Doc Text:
Cause: Spice needs some time at the end of migration to transfer internal state to destination. However, it needs helping hand from libvirt. Consequence: With previous code it was possible to kill source qemu (and spice server) before internal state was transmitted. This lead into unresponsive client on destination side. Fix: After qemu reports migration has finished, libvirt waits for spice server to migrate. Libvirt checks repeatedly on the monitor until the spice server reports success. Qemu is not terminated meanwhile. Result: Spice client won't get unresponsive anymore.
Clone Of: 836133
Environment:
Last Closed: 2013-02-21 07:18:11 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
libvirtd.log (8.24 MB, text/plain)
2012-10-10 08:25 UTC, zhe peng 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:0276 0 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2013-02-20 21:18:26 UTC

Description Yonit Halperin 2012-06-28 07:48:52 UTC 
+++ This bug was initially created as a clone of Bug #836133 +++

Description of problem:
In order to support seamless migration (bug 836123) Spice needs to migrate its state, from the migration src to the target, when migration is completed. However, when libvirt identifies that migration has completed, it closes the src qemu, preventing spice from completing its migration.
We want to add a QMP event for signaling libvirt when spice migration has completed. libvirt will need to wait for this event before closing the src.
(spice api already contains a migration_completion callback for qemu).

In order for spice to recognize whether the installed libvirt supports this qmp event, we will also need to add a "seamless-migration=on" parameter to spice arguments in qemu. Only if this parameter is set, spice will execute the seamless migration pathway.
Comment 2 Yonit Halperin 2012-07-24 09:05:06 UTC 
Hi Michal,
In https://bugzilla.redhat.com/show_bug.cgi?id=836133#c6, Luiz raised a concern that a QMP event triggered by spice is not enough. What do you think? Are there any other requirements?

Thanks,
Yonit.
Comment 3 Dave Allan 2012-07-24 15:41:17 UTC 
Yonit, Michal is on vacation this week, do you need an immediate answer?
Comment 4 Michal Privoznik 2012-07-30 08:11:26 UTC 
Yonit,

I think we can extend the output of 'query-migrate' command by a boolean called 'spice-migration-complete', for instance. Because libvirt uses this command to poll on qemu like every 50ms. Like Luiz said, an event is a kind of tricky as libvirt might miss it. The scenario looks like this: with libvirt running an user starts an migration. Then he kills libvirtd and start it again after while. If qemu would send an event meanwhile it won't reach libvirt. However, when libvirt starts the second time, it knows there was a migration running and starts to poll again.
Comment 5 Yonit Halperin 2012-08-01 11:29:30 UTC 
(In reply to comment #4)
> Yonit,
> 
> I think we can extend the output of 'query-migrate' command by a boolean
> called 'spice-migration-complete', for instance. Because libvirt uses this
ok, I'll send a patch to qemu. I hope it will be accepted.

> command to poll on qemu like every 50ms. Like Luiz said, an event is a kind
> of tricky as libvirt might miss it. The scenario looks like this: with
> libvirt running an user starts an migration. Then he kills libvirtd and
> start it again after while. If qemu would send an event meanwhile it won't
> reach libvirt. However, when libvirt starts the second time, it knows there
> was a migration running and starts to poll again.
Comment 6 Dave Allan 2012-08-29 15:50:49 UTC 
Martin, can we get QA ack?
Comment 7 Min Zhan 2012-08-30 02:32:24 UTC 
(In reply to comment #6)
> Martin, can we get QA ack?

QA ack is given. Remove needinfo flag.
Comment 8 Michal Privoznik 2012-09-13 17:06:36 UTC 
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2012-September/msg00941.html
Comment 9 Michal Privoznik 2012-09-20 14:52:33 UTC 
Yet another patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2012-September/msg01417.html
Comment 10 Michal Privoznik 2012-09-26 11:37:13 UTC 
Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2012-September/msg00695.html
Comment 15 zhe peng 2012-10-10 08:25:00 UTC 
Created attachment 624651 [details]
libvirtd.log

libvirtd.log
Comment 17 zhe peng 2012-10-10 08:55:07 UTC 
Hi Michal:
    So quickly :), very thanks!

verify with build:
libvirt-0.10.2-2.el6.x86_64
qemu-kvm-0.12.1.2-2.320.el6.x86_64
spice-server-0.12.0-1.el6.x86_64
spice-gtk-0.14-2.el6.x86_64
spice-protocol-0.12.2-1.el6.noarch


step:
 1: prepare two machine for migrate 
 2: on source, create a windows guest with spice
 .....
  <graphics type='spice' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
 .....
 3: migrate guest to target
  #virsh migrate win7_64 qemu+ssh://target.redhat.com/system
 4: check qemu process,make sure seamless-migration=on
 #ps -ef | grep win7_64
qemu      8278     1 53 17:47 ?        00:26:51 /usr/libexec/qemu-kvm -name win7_64 -S -M rhel6.2.0 -enable-kvm -m 2048 -smp 4,sockets=4,cores=1,threads=1 -uuid d551e24f-1368-6ffb-02fd-01eb28301ab5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win7_64.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -drive file=/var/lib/libvirt/migrate/win7-64-virtio.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:38:7e:d1,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5906,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device qxl,id=video1,vram_size=67108864,bus=pci.0,addr=0x7 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 

 5: after migrate finished, check libvirt log
.....
61634 2012-10-10 21:40:18.071+0000: 1751: debug : qemuMonitorJSONIOProcessLine:166 : QEMU_MONITOR_RECV_REPLY: mon=0x7f53ac0cf700 reply={"return": {"status": "completed"}, "id": "libvirt-95"}
.....
61668 2012-10-10 21:40:18.071+0000: 1756: debug : virJSONValueToString:1133 : result={"execute":"query-spice","id":"libvirt-96"}
61669 2012-10-10 21:40:18.071+0000: 1756: debug : qemuMonitorJSONCommandWithFd:259 : Send command '{"execute":"query-spice","id":"libvirt-96"}' for write with FD -1
.....
61763 2012-10-10 21:40:18.073+0000: 1751: debug : qemuMonitorIOProcess:354 : QEMU_MONITOR_IO_PROCESS: mon=0x7f53ac0cf700 buf={"return": {"migrated": tr      ue, "enabled": true, "auth": "none", "port": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}^M
61764  len=134
61765 2012-10-10 21:40:18.073+0000: 1751: debug : qemuMonitorJSONIOProcessLine:146 : Line [{"return": {"migrated": true, "enabled": true, "auth": "none      ", "port": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}]
61766 2012-10-10 21:40:18.073+0000: 1751: debug : virJSONValueFromString:975 : string={"return": {"migrated": true, "enabled": true, "auth": "none", "p      ort": 5906, "host": "0.0.0.0", "channels": []}, "id": "libvirt-96"}

verification passed.
Comment 18 Yonit Halperin 2012-10-18 06:30:21 UTC 
Hi zhe,

Have you verified the bug fix with a remote viewer connection? Spice session migration is relevant only with a spice client session. According to the report in Bug 836133 comment #21, there is still a problem.
Comment 19 zhe peng 2012-10-18 07:10:24 UTC 
Hi Yonit,

    No,when verify this bug, i dont use virt-viewer or remote-viewer to connect guest. I try it yesterday, use virt-viewer to connect guest then doing migrate,i always got error like "{"migrated": false," in libvirt log, but migrate can complete,  I will follow up bug: https://bugzilla.redhat.com/show_bug.cgi?id=836133 to track the issue.
Comment 22 errata-xmlrpc 2013-02-21 07:18:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html
Note You need to log in before you can comment on or make changes to this bug.