Bug 1037455

Summary: Migration from RHEL6.5 host to RHEL7.0 host with spice failed (aborted)
Product: Red Hat Enterprise Linux 7 Reporter: Qunfang Zhang <qzhang>
Component: spiceAssignee: Default Assignee for SPICE Bugs <rh-spice-bugs>
Status: CLOSED DUPLICATE QA Contact: Desktop QE <desktop-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: acathrow, cfergeau, hhuang, juzhang, kraxel, mazhang, michen, qzhang, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-03 14:57:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Qunfang Zhang 2013-12-03 08:28:22 UTC
Description of problem:
Boot a guest with spice and qxl on RHEL6.5 released host, and then migrate to RHEL7.0 host with the same command line (both host with -M rhel6.5.0 machine type). Migration failed and qemu-kvm aborted. 

VNC has no problem. 

Version-Release number of selected component (if applicable):
RHEL6.5:
kernel-2.6.32-431.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.415.el6_5.3.x86_64
spice-server-0.12.4-6.el6.x86_64

RHEL7.0:
kernel-3.10.0-57.el7.x86_64
qemu-kvm-1.5.3-20.el7.x86_64
spice-server-0.12.4-3.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Boot up a guest on RHEL6.5 host with spice:

(gdb) r -cpu SandyBridge -M rhel6.5.0 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -name rhel7.0-64 -uuid 9a0e67ec-f286-d8e7-0548-0c1c9ec93009 -nodefconfig -nodefaults -monitor stdio -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -drive file=gluster://10.66.**.**/gv0/rhel7.0-64-qzhang.raw,if=none,id=drive-disk0,format=raw,cache=none,id=disk0 -device scsi-hd,drive=drive-disk0,bus=scsi0.0,id=scsi-disk0,bootindex=1  -drive file=gluster://10.66.**.**/gv0/boot.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device scsi-cd,drive=drive-ide0-1-0,bus=scsi0.0,id=scsi-cdrom -netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d5:51:8a,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=port1,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=port2,bus=virtio-serial0.0,id=port2 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6  -drive if=none,id=drive-fdc0-0-0,format=raw,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -qmp tcp:0:5555,server,nowait -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -drive file=gluster://10.66.**.**/gv0/disk1.qcow2,if=none,id=disk1,format=qcow2,cache=none,id=disk1 -device virtio-blk-pci,bus=pci.0,addr=0x9,drive=disk1,id=disk1 -drive file=gluster://10.66.**.**/gv0/disk2.qcow2,if=none,id=disk2,format=qcow2,cache=none,id=disk2 -device ide-drive,bus=ide.0,unit=1,drive=disk2,id=disk2  -netdev tap,id=hostnet1,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device e1000,netdev=hostnet1,id=net1,mac=52:54:00:d5:51:2a,bus=pci.0,addr=0xa -netdev tap,id=hostnet2,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device e1000,netdev=hostnet2,id=net2,mac=52:54:00:d5:51:3a,bus=pci.0,addr=0xb 


2. Boot up the guest on destination host with "-incoming tcp:0:5800"

3. Migrate the guest from rhel6.5 host to rhel7.0 host. 

Actual results:
Guest aborted on destination host. 

Expected results:
Guest should work well on destination host after migration.

Additional info:

Program received signal SIGABRT, Aborted.
0x00007ffff30db979 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff30db979 in raise () from /lib64/libc.so.6
#1  0x00007ffff30dd088 in abort () from /lib64/libc.so.6
#2  0x00007ffff3e9051c in spice_logv (log_domain=0x7ffff3f061c6 "Spice", log_level=SPICE_LOG_LEVEL_ERROR, 
    strloc=0x7ffff3f06569 "char_device.c:937", 
    function=0x7ffff3f06c60 <__FUNCTION__.21514> "spice_char_device_state_restore", 
    format=0x7ffff3f0619e "assertion `%s' failed", args=args@entry=0x7fffffffd188) at log.c:109
#3  0x00007ffff3e90668 in spice_log (log_domain=log_domain@entry=0x7ffff3f061c6 "Spice", 
    log_level=log_level@entry=SPICE_LOG_LEVEL_ERROR, strloc=strloc@entry=0x7ffff3f06569 "char_device.c:937", 
    function=function@entry=0x7ffff3f06c60 <__FUNCTION__.21514> "spice_char_device_state_restore", 
    format=format@entry=0x7ffff3f0619e "assertion `%s' failed") at log.c:123
#4  0x00007ffff3e384da in spice_char_device_state_restore (dev=0x5555569b9db0, mig_data=0x5555569b9320)
    at char_device.c:937
#5  0x00007ffff3e7217b in reds_agent_state_restore (mig_data=<optimized out>) at reds.c:1364
#6  0x00007ffff3e75835 in attach_to_red_agent (sin=0x555556502bf8) at reds.c:3594
#7  spice_server_char_device_add_interface (sin=0x555556502bf8, s=0x555556504810) at reds.c:3698
#8  spice_server_add_interface (s=0x555556504810, sin=sin@entry=0x555556502bf8) at reds.c:3836
#9  0x0000555555749995 in qemu_spice_add_interface (sin=sin@entry=0x555556502bf8) at ui/spice-core.c:831
#10 0x0000555555740ebb in vmc_register_interface (scd=0x555556502bf0) at spice-qemu-char.c:124
#11 0x000055555578b7fe in virtio_serial_post_load_timer_cb (opaque=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/hw/char/virtio-serial-bus.c:602
#12 0x0000555555728156 in qemu_run_timers (clock=0x5555564f9f30) at qemu-timer.c:394
#13 0x00005555557283f5 in qemu_run_timers (clock=<optimized out>) at qemu-timer.c:459
#14 qemu_run_all_timers () at qemu-timer.c:452
#15 0x00005555556f608e in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:470
#16 0x00005555555ff3f8 in main_loop () at vl.c:1986
#17 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4343
(gdb)

Comment 1 Qunfang Zhang 2013-12-03 08:32:34 UTC
Not sure whether it is relevant, but the "info qtree" result for qxl is not the same between RHEL6.5 host and RHEL7.0 host. The command line on the two hosts are the same, and both using "-M rhel6.5.0"

On rhel 6.5:

 dev: qxl-vga, id ""
        dev-prop: ram_size = 67108864
        dev-prop: vram_size = 67108864
        dev-prop: revision = 4
        dev-prop: debug = 0
        dev-prop: guestdebug = 0
        dev-prop: cmdlog = 0
        bus-prop: addr = 02.0
        bus-prop: romfile = "vgabios-qxl.bin"
        bus-prop: rombar = 1
        bus-prop: multifunction = off
        class VGA controller, addr 00:02.0, pci id 1b36:0100 (sub 1af4:1100)
        bar 0: mem at 0xffffffffffffffff [0x3fffffe]
        bar 1: mem at 0xffffffffffffffff [0x3fffffe]
        bar 2: mem at 0xffffffffffffffff [0x1ffe]
        bar 3: i/o at 0xffffffffffffffff [0x1e]
        bar 6: mem at 0xffffffffffffffff [0xfffe]

On rhel7.0:

 dev: qxl-vga, id ""
        ram_size = 67108864
        vram_size = 67108864
        revision = 4
        debug = 0
        guestdebug = 0
        cmdlog = 0
        ram_size_mb = 4294967295
        vram_size_mb = 4294967295
        vram64_size_mb = 4294967295
        vgamem_mb = 16
        surfaces = 1024
        addr = 02.0
        romfile = "vgabios-qxl.bin"
        rombar = 1
        multifunction = off
        command_serr_enable = off
        class VGA controller, addr 00:02.0, pci id 1b36:0100 (sub 1af4:1100)
        bar 0: mem at 0xffffffffffffffff [0x3fffffe]
        bar 1: mem at 0xffffffffffffffff [0x3fffffe]
        bar 2: mem at 0xffffffffffffffff [0x1ffe]
        bar 3: i/o at 0xffffffffffffffff [0x1e]
        bar 6: mem at 0xffffffffffffffff [0xfffe]

Comment 3 Gerd Hoffmann 2013-12-03 14:29:21 UTC
Stack trace looks like the bug is in the spice agent channel setup.
Failed assert in spice_char_device_state_restore().
Assigning to spice-server for investigation.
Hmm?  No spice-server component?  Picking spice instead.

Does it work if you configure the machine without spicevmc?
Is a client connected while migrating?
Do you use seamless migration?

Comment 4 Christophe Fergeau 2013-12-03 14:57:59 UTC
This is a duplicae of rhbz#1035184

*** This bug has been marked as a duplicate of bug 1035184 ***

Comment 5 Qunfang Zhang 2013-12-04 03:22:01 UTC
(In reply to Gerd Hoffmann from comment #3)
> Stack trace looks like the bug is in the spice agent channel setup.
> Failed assert in spice_char_device_state_restore().
> Assigning to spice-server for investigation.
> Hmm?  No spice-server component?  Picking spice instead.
> 
> Does it work if you configure the machine without spicevmc?
> Is a client connected while migrating?
> Do you use seamless migration?

Seems the problems only occurs when "seamless-migration=on"+"withspicevmc"+"login guest". Please check the following detail info:

spice client connected,seamless-migration=on:

1. with spicevmc, do not login guest: pass.
2. with spicevmc, login guest:  fail.
3. without spicevmc, do not login guest: pass.
4. without spicevmc, login guest: pass.

spice client disconnected,seamless-migration=off:
5. with spicevmc, login guest: pass

spice client connected, seamless-migration=off:
6. withspicevmc,login guest: pass.

Comment 6 Qunfang Zhang 2013-12-04 03:23:28 UTC
(In reply to Qunfang Zhang from comment #5)
> (In reply to Gerd Hoffmann from comment #3)
> > Stack trace looks like the bug is in the spice agent channel setup.
> > Failed assert in spice_char_device_state_restore().
> > Assigning to spice-server for investigation.
> > Hmm?  No spice-server component?  Picking spice instead.
> > 
> > Does it work if you configure the machine without spicevmc?
> > Is a client connected while migrating?
> > Do you use seamless migration?
> 
> Seems the problems only occurs when
> "seamless-migration=on"+"withspicevmc"+"login guest". Please check the
> following detail info:
> 
> spice client connected,seamless-migration=on:
> 
> 1. with spicevmc, do not login guest: pass.
> 2. with spicevmc, login guest:  fail.
> 3. without spicevmc, do not login guest: pass.
> 4. without spicevmc, login guest: pass.
> 
> spice client disconnected,seamless-migration=off:
Should be: spice client disconnected,seamless-migration=on

> 5. with spicevmc, login guest: pass
> 
> spice client connected, seamless-migration=off:
> 6. withspicevmc,login guest: pass.