Bug 1150163

Summary: VM goes paused when connecting using SPICE after migration RHEL7.0 -> RHEL 6.5
Product: Red Hat Enterprise Linux 7 Reporter: Francesco Romani <fromani>
Component: qemu-kvmAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED CANTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: dgilbert, juzhang, rbalakri, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-07 15:28:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirt debug logs on destination (RHEL 6.5) host none

Description Francesco Romani 2014-10-07 14:40:45 UTC 
Description of problem:
First and foremost, I'm filing against RHEL7 because migrations between RHEL 6.x works fine. Not sure is RHEL7 bug or RHEL6 bug, however.

While reproducing a different issue using oVirt/libvirt, I noticed that after
a migration from RHEL 7.0 -> RHEL 6.5 reported as succesfull, when I try to connect to the VM on RHEL 6.5 host, it gets paused.

QEMU log reports:
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 1.732000 ms, bitrate 531258106 bps (506.647211 Mbps)
red_dispatcher_set_cursor_peer: 
inputs_connect: inputs channel client create
kvm: unhandled exit 80000021
kvm_run returned -22


SPICE console remain blacks


Version-Release number of selected component (if applicable):
RHEL 7:
BENji> 16:29:28 root [~]$ rpm -qa | grep qemu
qemu-kvm-rhev-1.5.3-60.el7_0.2.x86_64
ipxe-roms-qemu-20130517-5.gitc4bce43.el7.noarch
qemu-kvm-tools-rhev-1.5.3-60.el7_0.2.x86_64
qemu-kvm-common-rhev-1.5.3-60.el7_0.2.x86_64
qemu-img-rhev-1.5.3-60.el7_0.2.x86_64
libvirt-daemon-driver-qemu-1.1.1-29.el7_0.3.x86_64
BENji> 16:37:43 root [~]$ rpm -qa | grep spice
spice-server-0.12.4-5.el7.x86_64

RHEL 6:
TOji> 16:36:56 root [~]$ rpm -qa | grep qemu
qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64
gpxe-roms-qemu-0.9.7-6.10.el6.noarch
qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.14.x86_64
TOji> 16:37:36 root [~]$ rpm -qa | grep spice
spice-gtk-0.20-11.el6_5.1.x86_64
spice-gtk-python-0.20-11.el6_5.1.x86_64
spice-glib-0.20-11.el6_5.1.x86_64
spice-server-0.12.4-6.el6_5.2.x86_64


How reproducible:
Not sure 100% of times, but it happened three times in a row

Steps to Reproduce:
1. start a VM on RHEL 7 host using commandline reported below
2. move it to RHEL 6.5
3. connect to VM using SPICE once migration succeeds

Actual results:
VM goes paused


Expected results:
VM continues to run


Additional info:
full QEMU logs.

src (RHEL 7.0)
2014-10-07 14:32:18.209+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name FBD_1 -S -machine rhel6.5.0,accel=kvm,usb=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid d9be4550-7af1-4580-9430-4bfdaa7d50cf -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7.0-1.el7,serial=4C4C4544-0058-4410-8043-B5C04F4E5A31,uuid=d9be4550-7af1-4580-9430-4bfdaa7d50cf -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/FBD_1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2014-10-07T14:32:18,driftfix=slew -no-kvm-pit-reinjection -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x6 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-000000000064/4ac5f0af-b10e-41ee-9322-2e523f390306/images/ba737308-0fca-4c6d-8f0f-42a6e762c555/2f07f607-6ff0-4f90-9341-6242201c0221,if=none,id=drive-virtio-disk0,format=qcow2,serial=ba737308-0fca-4c6d-8f0f-42a6e762c555,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/rhev/data-center/00000002-0002-0002-0002-000000000064/4ac5f0af-b10e-41ee-9322-2e523f390306/images/3b12ce02-d9da-46f8-a743-b3937bf5a6d2/e239621b-a23d-4e6e-986e-e81d1aee7f6e,if=none,id=drive-virtio-disk1,format=qcow2,serial=3b12ce02-d9da-46f8-a743-b3937bf5a6d2,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x13,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/rhev/data-center/00000002-0002-0002-0002-000000000064/4ac5f0af-b10e-41ee-9322-2e523f390306/images/5180f292-ea83-44ec-82f5-bc3aed59384c/cf786231-bc69-432f-909c-d17660f2f7fc,if=none,id=drive-virtio-disk2,format=qcow2,serial=5180f292-ea83-44ec-82f5-bc3aed59384c,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x14,drive=drive-virtio-disk2,id=virtio-disk2 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:d8:81:01,bus=pci.0,addr=0x3,bootindex=2 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/d9be4550-7af1-4580-9430-4bfdaa7d50cf.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/d9be4550-7af1-4580-9430-4bfdaa7d50cf.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=192.168.1.39,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=33554432 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8
Warning: option deprecated, use lost_tick_policy property of kvm-pit instead.
2014-10-07 14:33:10.360+0000: shutting down
qemu: terminating on signal 15 from pid 11231


dst (RHEL 6.5)
2014-10-07 14:32:45.462+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name FBD_1 -S -M rhel6.5.0 -cpu SandyBridge -enable-kvm -m 4096 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid d9be4550-7af1-4580-9430-4bfdaa7d50cf -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7.0-1.el7,serial=4C4C4544-0058-4410-8043-B5C04F4E5A31,uuid=d9be4550-7af1-4580-9430-4bfdaa7d50cf -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/FBD_1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2014-10-07T14:32:45,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x6 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-000000000064/4ac5f0af-b10e-41ee-9322-2e523f390306/images/ba737308-0fca-4c6d-8f0f-42a6e762c555/2f07f607-6ff0-4f90-9341-6242201c0221,if=none,id=drive-virtio-disk0,format=qcow2,serial=ba737308-0fca-4c6d-8f0f-42a6e762c555,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/rhev/data-center/00000002-0002-0002-0002-000000000064/4ac5f0af-b10e-41ee-9322-2e523f390306/images/3b12ce02-d9da-46f8-a743-b3937bf5a6d2/e239621b-a23d-4e6e-986e-e81d1aee7f6e,if=none,id=drive-virtio-disk1,format=qcow2,serial=3b12ce02-d9da-46f8-a743-b3937bf5a6d2,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x13,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/rhev/data-center/00000002-0002-0002-0002-000000000064/4ac5f0af-b10e-41ee-9322-2e523f390306/images/5180f292-ea83-44ec-82f5-bc3aed59384c/cf786231-bc69-432f-909c-d17660f2f7fc,if=none,id=drive-virtio-disk2,format=qcow2,serial=5180f292-ea83-44ec-82f5-bc3aed59384c,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x14,drive=drive-virtio-disk2,id=virtio-disk2 -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:d8:81:01,bus=pci.0,addr=0x3,bootindex=2 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/d9be4550-7af1-4580-9430-4bfdaa7d50cf.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/d9be4550-7af1-4580-9430-4bfdaa7d50cf.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice port=5900,tls-port=5901,addr=192.168.1.48,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=33554432 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -incoming tcp:[::]:49156 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 1.732000 ms, bitrate 531258106 bps (506.647211 Mbps)
red_dispatcher_set_cursor_peer: 
inputs_connect: inputs channel client create
kvm: unhandled exit 80000021
kvm_run returned -22
Comment 2 Francesco Romani 2014-10-07 14:48:49 UTC 
Created attachment 944616 [details]
libvirt debug logs on destination (RHEL 6.5) host

added libvirt debug logs for destination (6.5) host. Relevant because we can track monitor traffic to qemu using these logs.
Comment 3 Dr. David Alan Gilbert 2014-10-07 15:28:03 UTC 
Sorry, Migration 7.x->6.x is not supported