RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1691701 - rhel6 guest reboot after migrate from rhel8 host to destination host
Summary: rhel6 guest reboot after migrate from rhel8 host to destination host
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: spice
Version: 8.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.0
Assignee: Uri Lublin
QA Contact: SPICE QE bug list
URL:
Whiteboard:
: 1692212 (view as bug list)
Depends On:
Blocks: 1691721 1692212
TreeView+ depends on / blocked
 
Reported: 2019-03-22 09:50 UTC by jingzhao
Modified: 2020-11-14 06:16 UTC (History)
16 users (show)

Fixed In Version: spice-0.14.2-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-05 20:59:07 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3392 0 None None None 2019-11-05 20:59:26 UTC

Description jingzhao 2019-03-22 09:50:57 UTC 
Description of problem:
rhel6 guest reboot after migrate from rhel8 host to rhel7.6 host
(seems rhel6 guest logout and you must login use the usr and password)

Version-Release number of selected component (if applicable):
source host(rhel7.6):
kernel-3.10.0-957.12.1.el7.x86_64
qemu-kvm-1.5.3-160.el7_6.1.x86_64.rpm or qemu-kvm-rhev-2.12.0-18.el7_6.3.x86_64

Destination host(rhel8):
kernel-4.18.0-80.el8.x86_64
qemu-kvm-3.1.0-20.module+el8+2888+cdc893a8.x86_64

How reproducible:
3/3

Steps to Reproduce:
1. Boot rhel6 guest with qemu command[1]

2. Migrate from rhel7.6 host to rhel8 host and then migrate from rhel8 host to rhel7.6 guest


Actual results:
rhel6 guest will reboot after migrate from rhel8 to rhel7.6 finished
(seems logout and you must login with usr and password)

Expected results:
rhel6 guest didn't reboot


Additional info:
[1]
/usr/libexec/qemu-kvm \
-name rhel7 \
-machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off \
-realtime mlock=off \
-cpu SandyBridge,+x2apic,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic,hv_time \
-sandbox off \
-m 7680   \
-smp 8,cores=1,threads=1,sockets=8  \
-uuid 49a3438a-70a3-4ba8-92ce-3a05e0934608 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,path=/mnt/tmp1,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-no-hpet \
-no-shutdown \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot order=c,menu=on,splash-time=3000,strict=on \
-device ich9-usb-ehci1,id=usb0,bus=pci.0,addr=0x5.0x7 \
-device ich9-usb-uhci1,masterbus=usb0.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 \
-device ich9-usb-uhci2,masterbus=usb0.0,firstport=2,bus=pci.0,addr=0x5.0x1 \
-device ich9-usb-uhci3,masterbus=usb0.0,firstport=4,bus=pci.0,addr=0x5.0x2 \
-device virtio-scsi-pci,id=scsi0,cmd_per_lun=234,bus=pci.0,addr=0x8 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 \
-drive file=/mnt/virtio-win-1.9.4.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw \
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 \
-drive file=/mnt/virtio-win-1.9.6_x86.vfd,if=none,id=drive-fdc0-0-0,format=raw,cache=none \
-global isa-fdc.driveA=drive-fdc0-0-0 \
-drive file=/mnt/rhel76-64-virtio-scsi.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,discard=unmap,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-drive file=/mnt/virtio-scsi-disk,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-scsi-pci,id=scsi1,addr=0x13 \
-device scsi-hd,drive=drive-scsi-disk,bus=scsi1.0,id=data-disk2 \
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-chardev socket,id=charchannel1,path=/mnt/tmp2,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0 \
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-device intel-hda,id=sound1,bus=pci.0,addr=0x6 \
-device hda-micro,id=sound1-codec0,bus=sound1.0 \
-device intel-hda,id=sound2,bus=pci.0,addr=0x17 \
-device hda-output,id=sound2-codec0,bus=sound2.0,cad=0 \
-device ich9-intel-hda,id=sound3,bus=pci.0,addr=0x18 \
-device hda-duplex,id=sound3-codec0,bus=sound3.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1 \
-chardev spicevmc,id=charredir2,name=usbredir \
-device usb-redir,chardev=charredir2,id=redir2 \
-chardev spicevmc,id=charredir3,name=usbredir \
-device usb-redir,chardev=charredir3,id=redir3 \
-device usb-host,id=hostdev0 \
-device pvpanic,ioport=1285 -msg timestamp=on \
-netdev tap,id=hostnet0,vhost=on,id=hostnet0,script=/etc/qemu-ifup \
-device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=b6:af:42:c8:66:18,bus=pci.0,addr=0x14 \
-netdev tap,id=hostnet1,vhost=on,script=/etc/qemu-ifup  \
-device e1000,netdev=hostnet1,id=virtio-net-pci1,mac=b6:2f:a8:85:82:7c,bus=pci.0,addr=0x15,multifunction=off \
-netdev tap,id=hostnet2,vhost=on,script=/etc/qemu-ifup \
-device rtl8139,netdev=hostnet2,id=virtio-net-pci2,mac=4e:63:28:bc:c1:75,bus=pci.0,addr=0x16,multifunction=off \
-drive file=/mnt/ide-disk,if=none,id=drive-data-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,copy-on-read=off,media=disk \
-device ide-hd,drive=drive-data-disk,id=system-disk,logical_block_size=512,physical_block_size=512,min_io_size=32,opt_io_size=64,discard_granularity=512,ver=fuxc-ver,bus=ide.0,unit=0  \
-device ich9-usb-uhci6,id=uhci6,bus=pci.0,addr=0xa \
-device usb-kbd,id=kdb0,bus=uhci6.0 \
-device ich9-usb-uhci5,id=uhci5,bus=pci.0,addr=0xb \
-device usb-mouse,id=mouse0,bus=uhci5.0 \
-device nec-usb-xhci,id=xhci,bus=pci.0,addr=0xd \
-device usb-bot,id=bot1,bus=xhci.0 \
-drive file=/mnt/virtio-win-126.iso,if=none,id=usb-cdrom1,format=raw \
-device scsi-cd,bus=bot1.0,scsi-id=0,lun=1,drive=usb-cdrom1,id=usb-cdrom1 \
-drive file=/mnt/bot-disk1,id=usb-disk1,if=none,format=qcow2 \
-device scsi-hd,bus=bot1.0,scsi-id=0,lun=0,drive=usb-disk1,id=usb-disk1 \
-device usb-ehci,id=ehci,bus=pci.0,addr=0xe \
-device usb-bot,id=bot2,bus=ehci.0 \
-drive file=/mnt/virtio-win-126-cp1.iso,if=none,id=usb-cdrom2,format=raw \
-device scsi-cd,bus=bot2.0,scsi-id=0,lun=1,drive=usb-cdrom2,id=usb-cdrom2 \
-drive file=/mnt/bot-disk2,id=usb-disk2,if=none,format=qcow2 \
-device scsi-hd,bus=bot2.0,scsi-id=0,lun=0,drive=usb-disk2,id=usb-disk2 \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0xf \
-device usb-bot,id=bot3,bus=usb.0 \
-drive file=/mnt/virtio-win-126-cp2.iso,if=none,id=usb-cdrom3,format=raw \
-device scsi-cd,bus=bot3.0,scsi-id=0,lun=1,drive=usb-cdrom3,id=usb-cdrom3 \
-drive file=/mnt/bot-disk3,id=usb-disk3,if=none,format=qcow2 \
-device scsi-hd,bus=bot3.0,scsi-id=0,lun=0,drive=usb-disk3,id=usb-disk3 \
-device ich9-usb-uhci3,id=uhci,bus=pci.0,addr=0x10 \
-device usb-storage,drive=drive-usb-0,id=usb-0,removable=on,bus=uhci.0,port=1 \
-drive file=/mnt/usb-uhci,if=none,id=drive-usb-0,media=disk,format=qcow2 \
-device ich9-usb-ehci1,id=ehci1,bus=pci.0,addr=0x11 \
-device usb-storage,drive=drive-usb-1,id=usb-1,removable=on,bus=ehci1.0,port=1 \
-drive file=/mnt/usb-ehci,if=none,id=drive-usb-1,media=disk,format=qcow2 \
-device nec-usb-xhci,id=xhci1,bus=pci.0,addr=0x12 \
-device usb-storage,drive=drive-usb-2,id=usb-2,removable=on,bus=xhci1.0,port=1 \
-drive file=/mnt/usb-xhci,if=none,id=drive-usb-2,media=disk,format=qcow2 \
-watchdog ib700 -watchdog-action reset \
-device virtio-rng-pci,id=rng0,bus=pci.0,addr=0x1c \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x1d \
-monitor stdio \
-qmp tcp:0:4466,server,nowait -serial unix:/tmp/ttym,server,nowait \
-spice port=5910,addr=0.0.0.0,disable-ticketing,seamless-migration=on \
-k en-us \
-device qxl-vga,id=video0,ram_size=134217728,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 \
-chardev socket,id=seabioslog_id,path=/home/seabios,server,nowait \
-device isa-debugcon,chardev=seabioslog_id,iobase=0x402 \


[2] note: pc-i440fx-7.0.0 machine type when using qemu-kvm-1.5.3-160.el7_6.1.x86_64.rpm

[3] rhel6 guest work well and didn't reboot when rhel8 host using qemu-kvm-2.12.0-63.module+el8+2833+c7d6d092.x86_64
Comment 1 jingzhao 2019-03-22 09:53:41 UTC 
Didn't reproduce the issue when I used RHEL.7.6 guest on above scenario
Comment 2 Pei Zhang 2019-03-22 09:57:07 UTC 
Migrate RHEL6.10 guest from RHEL8.0 src host - > RHEL8.0 des host also hit this issue:
  - PC: fail
  - Q35: fail 

Versions: qemu-kvm-3.1.0-20.module+el8+2888+cdc893a8.x86_64
Comment 3 Pei Zhang 2019-03-22 10:03:12 UTC 
(In reply to Pei Zhang from comment #2)
> Migrate RHEL6.10 guest from RHEL8.0 src host - > RHEL8.0 des host also hit
> this issue:
>   - PC: fail
>   - Q35: fail 
> 
> Versions: qemu-kvm-3.1.0-20.module+el8+2888+cdc893a8.x86_64

== Full cmdline with Q35:
/usr/libexec/qemu-kvm -name rhel6.10 \
-M q35,kernel-irqchip=split \
-cpu SandyBridge -m 4G \
-device intel-iommu,intremap=true,caching-mode=true \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,chassis=1 \
-device pcie-root-port,id=root.2,chassis=2 \
-device virtio-scsi-pci-transitional,id=virtio_scsi_pci0,bus=root.1 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/mnt/rhel610-64-virtio-scsi.qcow2 \
-device scsi-hd,drive=drive_image1,id=image1,bus=virtio_scsi_pci0.0 \
-vnc :2 \
-vga qxl \
-monitor stdio \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci-transitional,netdev=hostnet0,id=net0,mac=18:66:da:5f:d1:02,bus=root.2 \


== Full cmdline with PC:
/usr/libexec/qemu-kvm -name rhel6.10 \
-M pc \
-cpu SandyBridge -m 4G \
-smp 4,sockets=1,cores=4,threads=1 \
-device virtio-scsi-pci-transitional,id=virtio_scsi_pci0 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/mnt/rhel610-64-virtio-scsi.qcow2 \
-device scsi-hd,drive=drive_image1,id=image1,bus=virtio_scsi_pci0.0 \
-vnc :2 \
-vga qxl \
-monitor stdio \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci-transitional,netdev=hostnet0,id=net0,mac=18:66:da:5f:d1:02 \
Comment 5 Dr. David Alan Gilbert 2019-03-25 12:24:24 UTC 
I can reproduce it; for me the guest isn't actually rebooting, it's X in the guest that's crashing, so you see the GUI come back after
the X server restarts, but if you check the uptime it's not rebooted.
The X crash is always in the QXL driver seg faultin.

I've also seen qemu crash on the destination in spice, with a trace like:
(qemu) red_qxl_loadvm_commands:
id 0, group 0, virt start 0, virt end ffffffffffffffff, generation 0, delta 0
id 1, group 1, virt start 7fab5fc00000, virt end 7fab63bfe000, generation 0, delta 7fab5fc00000
id 2, group 1, virt start 7fab5ba00000, virt end 7fab5fa00000, generation 0, delta 7fab5ba00000

(process:3896): Spice-CRITICAL **: 12:00:05.506: memslot.c:122:memslot_get_virt: address generation is not valid, group_id 1, slot_id 0, gen 6, slot_gen 0

Thread 7 (Thread 0x7fac6ac1b700 (LWP 3905)):
#0  0x00007fac8a8a0b44 in read () at /lib64/libpthread.so.0
#1  0x00007fac8ba57c89 in spice_backtrace_gstack () at /lib64/libspice-server.so.1
#2  0x00007fac8ba5f270 in spice_log () at /lib64/libspice-server.so.1
#3  0x00007fac8ba24351 in memslot_get_virt () at /lib64/libspice-server.so.1
#4  0x00007fac8ba2cbc8 in red_get_data_chunks_ptr () at /lib64/libspice-server.so.1
#5  0x00007fac8ba2f2e4 in red_get_cursor_cmd () at /lib64/libspice-server.so.1
#6  0x00007fac8ba3fc5f in red_process_cursor_cmd () at /lib64/libspice-server.so.1
#7  0x00007fac8ba3fe0b in handle_dev_loadvm_commands () at /lib64/libspice-server.so.1
#8  0x00007fac8ba0d7a8 in dispatcher_handle_recv_read () at /lib64/libspice-server.so.1
#9  0x00007fac8ba141bf in watch_func () at /lib64/libspice-server.so.1
#10 0x00007fac8eba089d in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#11 0x00007fac8eba0c68 in g_main_context_iterate.isra () at /lib64/libglib-2.0.so.0
#12 0x00007fac8eba0f92 in g_main_loop_run () at /lib64/libglib-2.0.so.0
#13 0x00007fac8ba40efe in red_worker_main () at /lib64/libspice-server.so.1
#14 0x00007fac8a8972de in start_thread () at /lib64/libpthread.so.0
#15 0x00007fac8a5c7a63 in clone () at /lib64/libc.so.6


in the cases where X is crashing I've seen messages like:
(process:4437): Spice-WARNING **: 12:08:46.153: cursor-channel.c:277:cursor_channel_process_cmd: invalid cursor command 255
and
(process:23690): Spice-WARNING **: 12:17:35.665: cursor-channel.c:277:cursor_channel_process_cmd: invalid cursor command 151

so I bet this is one of the fixes for the bad cursor state.
Note, if X isn't running in the guest it's fine.
Comment 6 Dr. David Alan Gilbert 2019-03-25 13:07:28 UTC 
Trying upstream, 34.0.0-rc, 3.0.0, 2.12.0 all crashed in the same way when built on rhel 8
2.11.0 apparently works (needed a small hack to memfd to build)
Comment 7 Dr. David Alan Gilbert 2019-03-25 13:15:34 UTC 
4.0.0 rc works fine if built on rhel7
Hmm lots of differences.
spice-server on rhel8: spice-server-devel-0.14.0-7.el8.x86_64
spice-server on rhel7: spice-server-devel-0.14.0-6.el7.x86_64
Comment 8 Dr. David Alan Gilbert 2019-03-25 13:26:36 UTC 
rhel7 on 4.0.0rc is still fine even with spice-server 0.14.0-7.el7
Comment 9 Dr. David Alan Gilbert 2019-03-25 13:35:25 UTC 
hmm, except the bug has stopped failing for me at the moment. so not actually sure which cases work.
Comment 10 Dr. David Alan Gilbert 2019-03-25 17:26:13 UTC 
Note similarities to bz 1540919
Comment 11 Dr. David Alan Gilbert 2019-03-25 18:13:52 UTC 
The segs in the guest X server all seem to be in qxl_garbage_collect_internal, or qxl_mem.c it calls:

12:08:53-2142
backtrace in rhel6 guest:
qxl_mem.c:501 qxl_bo_map   _bo=0xffffff00fffffc
qxl_garbage_collect_internal
qxl_grabage_collect
qxl_allocnf
qx_blo_alloc_internal

16:33:05-2142
qxl_mem.c:501  _bo=0x5aaeaeae00fffffc
qxl_garbage_collect_internal
qxl_grabage_collect
qxl_allocatenf
qxl_bo_alloc_internal

11:46:12
qxl_garbage_collect_internal cmd=0x1609 and segs in cmd->type (info_bo = 0x55614fe3e1a0)
Comment 12 Dr. David Alan Gilbert 2019-03-25 18:44:09 UTC 
Christophe points out that rhel8's spice-server is missing some fo the fixes that are in the rhel7 spice server; so lets bounce it over to them.
(specifically ones from https://bugzilla.redhat.com/show_bug.cgi?id=1567944 )
Comment 14 Dr. David Alan Gilbert 2019-07-26 12:10:44 UTC 
Please retest with the latest spice-server packages installed on the destination (0.14.2-1 or newer)
If it's fixed then please also retest 1692212 and if that's also fixed mark as a dupe of this bz.
Comment 16 Dr. David Alan Gilbert 2019-07-31 08:38:24 UTC 
*** Bug 1692212 has been marked as a duplicate of this bug. ***
Comment 20 errata-xmlrpc 2019-11-05 20:59:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3392
Note You need to log in before you can comment on or make changes to this bug.