Bug 867724

Summary: Libvirt sometimes fails to wait on spice to migrate
Product: Red Hat Enterprise Linux 6 Reporter: Michal Privoznik <mprivozn>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4CC: acathrow, dyasny, dyuan, mzhan, rwu, weizhan, ydu, zhpeng, zpeng
Target Milestone: rc   
Target Release: 6.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.10.2-5.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 07:10:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 836133    
Attachments:
Description Flags
full libvirtd log none

Description Michal Privoznik 2012-10-18 07:56:35 UTC 
Description of problem:

Libvirt currently holds up migration and lets spice to migrate its internal data. During this, libvirt repeatedly asks 'query-spice' to determine if it has completed. Qemu replies with a boolean. However, libvirt expects string so the conversion will fail and libvirt simply fallbacks into pre-seamless mode by proclaiming migration as finished.

Version-Release number of selected component (if applicable):

libvirt-0.10.2-2.el6.x86_64

How reproducible:

few times, when spice hasn't migrated its state yet and 'false' is reported to 'query-spice'

Steps to Reproduce:
1.
2.
3.
  
Actual results:

libvirt prematurely kills qemu

Expected results:

libvirt kills qemu after query-spice reported 'true'.

Additional info:

Upstream patch has been proposed:
https://www.redhat.com/archives/libvir-list/2012-October/msg00961.html
Comment 2 Michal Privoznik 2012-10-18 08:23:45 UTC 
*** Bug 867737 has been marked as a duplicate of this bug. ***
Comment 3 Michal Privoznik 2012-10-18 08:34:55 UTC 
Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2012-October/msg00934.html
Comment 5 zhe peng 2012-10-25 08:25:17 UTC 
I can reproduce this with libvirt-0.10.2-2.el6.x86_64

verify with pkgs:
libvirt-0.10.2-5.el6.x86_64
kernel-2.6.32-323.el6.x86_64
qemu-kvm-0.12.1.2-2.320.el6.x86_64
spice-server-0.12.0-1.el6.x86_64
spice-gtk-0.14-4.el6.x86_64
spice-protocol-0.12.2-1.el6.noarch
spice-glib-0.14-4.el6.x86_64
virt-viewer-0.5.2-16.el6.x86_64

step:
 1: prepare two machine for migration 
 2: on source, create a windows guest with spice
 .....
  <graphics type='spice' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
 .....
 3: start guest ,use virt-viewer/remote-viewer to connect guest
  #virsh start $guest
  #virt-viewer $guest
  #remote-viewer spice://$source/?port=5901
 4: migrate guest to target
  #virsh migrate win7_64 qemu+ssh://target.redhat.com/system
 5: check qemu process,make sure seamless-migration=on
  #ps -ef | grep kvm
   .....
  -spice port=5901,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-US -vga qxl -global qxl-vga.vram_size=67108864 -device qxl,id=video1,vram_size=67108864,bus=pci.0,addr=0x7 -device intel-hda,id=sound0,bus=pci.0,addr=0x6
   .....
  6:after migrate finished,virt-viewer still connect the guest and worked well
  7:check libvirt log
 ...
 2012-10-25 21:42:11.089+0000: 27729: debug : virJSONValueToString:1133 :  result={"execute":"query-spice","id":"libvirt-123"}
 2012-10-25 21:42:11.089+0000: 27729: debug : qemuMonitorJSONCommandWithFd:263 : Send command '{"execute":"query-spice","id":"libvirt-123"}' for write with FD -1
   ......
   2012-10-25 21:42:11.090+0000: 27724: debug : qemuMonitorIOProcess:353 : QEMU_MONITOR_IO_PROCESS: mon=0x7fc6a4007250 buf={"return": {"migrated": false, "enabled": true, "auth": "none", "port": 5901, "host": "0.0.0.0", "channels": [{"port": "54078", "family": "ipv4", "channel-type": 1, "connection-id": 1804289383, "host": "127.0.0.1", "channel-id": 0, "tls": false}, {"port": "54083", "family": "ipv4", "channel-type": 9, "connection-id": 1804289383, "host": "127.0.0.1", "channel-id": 0, "tls": false}, {"port": "54084", "family": "ipv4", "channel-type": 2, "connection-id": 1804289383, "host": "127.0.0.1", "channel-id": 1, "tls": false}, {"port": "54089", "family": "ipv4", "channel-type": 4, "connection-id": 1804289383, "host": "127.0.0.1", "channel-id": 1, "tls": false}, {"port": "54088", "family": "ipv4", "channel-type": 3, "connection-id": 1804289383, "host": "127.0.0.1", "channel-id": 0, "tls": false}, {"port": "54085", "family": "ipv4", "channel-type": 2, "connection-id": 1804289383, "host": "127.0.0.1", "channel-id": 0, "tls": false}, {"port": "54091", "family": "ipv4", "channel-type": 4, "connection-id": 1804289383, "host": len=1023
  .....
 2012-10-25 21:42:20.543+0000: 27724: debug : qemuMonitorIOProcess:353 : QEMU_MONITOR_IO_PROCESS: mon=0x7fc6a4007250 buf={"return": {"migrated": true, "enabled": true, "auth": "none", "port": 5901, "host": "0.0.0.0", "channels": []}, "id": "libvirt-325"}^M
 len=135
2012-10-25 21:42:20.543+0000: 27724: debug : qemuMonitorJSONIOProcessLine:150 : Line [{"return": {"migrated": true, "enabled": true, "auth": "none", "port": 5901, "host": "0.0.0.0", "channels": []}, "id": "libvirt-325"}]
2012-10-25 21:42:20.543+0000: 27724: debug : virJSONValueFromString:975 : string={"return": {"migrated": true, "enabled": true, "auth": "none", "port": 5901, "host": "0.0.0.0", "channels": []}, "id": "libvirt-325"}
 ...
  
   Verification passed.
Comment 6 zhe peng 2012-10-25 08:28:10 UTC 
Created attachment 633223 [details]
full libvirtd log
Comment 7 errata-xmlrpc 2013-02-21 07:10:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html