Bug 1655963 - [RHEL8 fast train] src qemu core dump when do migration ( block device node-name changed after change cdrom)
Summary: [RHEL8 fast train] src qemu core dump when do migration ( block device node-n...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.0
Assignee: Dr. David Alan Gilbert
QA Contact: Xueqiang Wei
URL:
Whiteboard:
Depends On:
Blocks: 1655972 1659395
TreeView+ depends on / blocked
 
Reported: 2018-12-04 10:37 UTC by Xueqiang Wei
Modified: 2019-11-14 23:42 UTC (History)
13 users (show)

Fixed In Version: qemu 3.1.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1655972 1659395 (view as bug list)
Environment:
Last Closed: 2019-05-29 16:04:52 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
core dump log (21.21 KB, text/plain)
2018-12-04 11:17 UTC, Xueqiang Wei
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1293 0 None None None 2019-05-29 16:05:20 UTC

Description Xueqiang Wei 2018-12-04 10:37:57 UTC
Description of problem:

src qemu core dump when do migration.



Version-Release number of selected component (if applicable):

Host:
Kernel-4.18.0-37.el8.x86_64
qemu-kvm-3.0.0-2.module+el8+2246+78080371
virtio-win-prewhql-0.1-162.iso

Guest:
Windows2019


How reproducible:
5/5


Steps to Reproduce:
1. start src guest with below cmd lines. (blockdev+q35)
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_w2u90exl/monitor-qmpmonitor1-20181127-024837-wdAVx2FL,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_w2u90exl/monitor-catch_monitor-20181127-024837-wdAVx2FL,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idulvcka  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_w2u90exl/serial-serial0-20181127-024837-wdAVx2FL,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181127-024837-wdAVx2FL,path=/var/tmp/avocado_w2u90exl/seabios-20181127-024837-wdAVx2FL,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181127-024837-wdAVx2FL,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0 \
    -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2,node-name=my_file \
    -blockdev driver=qcow2,node-name=my,file=my_file \
    -device scsi-hd,drive=my,bus=virtio_scsi_pci0.0 \
    -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/images/data.qcow2,node-name=data_disk \
    -blockdev driver=qcow2,node-name=disk1,file=data_disk \
    -device scsi-hd,drive=disk1,bus=virtio_scsi_pci0.0 \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:34:35:36:37:38,id=idyb3F88,vectors=4,netdev=idTAFS0s,bus=pcie.0-root-port-7,addr=0x0  \
    -netdev tap,id=idTAFS0s,vhost=on \
    -m 4G  \
    -smp 12,maxcpus=12,cores=6,threads=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie.0-root-port-9,slot=9,chassis=9,addr=0x9,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie.0-root-port-9,addr=0x0 \
    -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,node-name=cd_disk1,read-only=on \
    -blockdev driver=raw,node-name=cd1,file=cd_disk1,read-only=on \
    -device scsi-cd,drive=cd1,bus=virtio_scsi_pci1.0,id=cd_disk1 \
    -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-162.iso,node-name=cd_disk2,read-only=on \
    -blockdev driver=raw,node-name=cd2,file=cd_disk2,read-only=on \
    -device scsi-cd,drive=cd2,bus=virtio_scsi_pci1.0,id=cd_disk2 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:4444,server,nowait \

2. start dst guest in listening mode (same cmd lines)
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_w2u90exl/monitor-qmpmonitor1-20181127-024837-wdAVx2FL,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_w2u90exl/monitor-catch_monitor-20181127-024837-wdAVx2FL,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idulvcka  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_w2u90exl/serial-serial0-20181127-024837-wdAVx2FL,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181127-024837-wdAVx2FL,path=/var/tmp/avocado_w2u90exl/seabios-20181127-024837-wdAVx2FL,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181127-024837-wdAVx2FL,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0 \
    -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2,node-name=my_file \
    -blockdev driver=qcow2,node-name=my,file=my_file \
    -device scsi-hd,drive=my,bus=virtio_scsi_pci0.0 \
    -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/images/data.qcow2,node-name=data_disk \
    -blockdev driver=qcow2,node-name=disk1,file=data_disk \
    -device scsi-hd,drive=disk1,bus=virtio_scsi_pci0.0 \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:34:35:36:37:38,id=idyb3F88,vectors=4,netdev=idTAFS0s,bus=pcie.0-root-port-7,addr=0x0  \
    -netdev tap,id=idTAFS0s,vhost=on \
    -m 4G  \
    -smp 12,maxcpus=12,cores=6,threads=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie.0-root-port-9,slot=9,chassis=9,addr=0x9,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie.0-root-port-9,addr=0x0 \
    -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,node-name=cd_disk1,read-only=on \
    -blockdev driver=raw,node-name=cd1,file=cd_disk1,read-only=on \
    -device scsi-cd,drive=cd1,bus=virtio_scsi_pci1.0,id=cd_disk1 \
    -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-162.iso,node-name=cd_disk2,read-only=on \
    -blockdev driver=raw,node-name=cd2,file=cd_disk2,read-only=on \
    -device scsi-cd,drive=cd2,bus=virtio_scsi_pci1.0,id=cd_disk2 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :1  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:4445,server,nowait \
    -incoming tcp:0:5800 \

3.  connect qmp monitor and query "cd2" info.
# telnet localhost 4444
{"execute":"qmp_capabilities"}
{"execute": "query-block"}

 {"device": "", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 567269376, "filename": "/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-162.iso", "format": "raw", "actual-size": 568389632, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "cd2", "backing_file_depth": 0, "drv": "raw", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-162.iso", "encryption_key_missing": false}, "qdev": "cd_disk2", "tray_open": false, "type": "unknown"}]}


4. eject cdrom, and check "cd2" info again

{"execute":"blockdev-open-tray","arguments":{"id":"cd_disk2"}}
{"execute":"blockdev-remove-medium","arguments":{"id":"cd_disk2"}}
{"execute": "query-block"}

{"device": "", "locked": false, "removable": true, "qdev": "cd_disk2", "tray_open": true, "type": "unknown"}

5. reload raw iso in cd2 before, and check block info again.

{"execute":"blockdev-change-medium","arguments":{"id": "cd_disk2","filename": "/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-162.iso", "format": "raw"}}

{"execute": "query-block"}

{"device": "", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 567269376, "filename": "/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-162.iso", "format": "raw", "actual-size": 568389632, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block171", "backing_file_depth": 0, "drv": "raw", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-162.iso", "encryption_key_missing": false}, "qdev": "cd_disk2", "tray_open": false, "type": "unknown"}

6. migrate from src to dst

{"execute": "migrate","arguments":{"uri": "tcp:0:5800"}}


Actual results:

after step 6, qemu core dump.

# sh cdrom_test_src.sh 
QEMU 3.0.0 monitor - type 'help' for more information
(qemu) c
(qemu) qemu-kvm: block.c:855: bdrv_child_cb_inactivate: Assertion `bs->open_flags & BDRV_O_INACTIVE' failed.
cdrom_test_src.sh: line 52:  6188 Aborted                 (core dumped)


# sh cdrom_test_dst.sh 
QEMU 3.0.0 monitor - type 'help' for more information
(qemu) qemu-kvm: Failed to load virtio_pci/modern_queue_state:avail
qemu-kvm: Failed to load virtio_pci/modern_state:vqs
qemu-kvm: Failed to load virtio/extra_state:extra_state
qemu-kvm: Failed to load virtio-scsi:virtio
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:09.0:00.0/virtio-scsi'
qemu-kvm: load of migration failed: Input/output error


Expected results:
Migrate successfully without any issue.



Additional info:

Migrate successfully without any cdrom operation.


And I found node-name changed from cd2 to #block171 after reload iso file.

"node-name": "cd2"
"node-name": "#block171"

Comment 1 Xueqiang Wei 2018-12-04 11:17:38 UTC
Created attachment 1511264 [details]
core dump log

Comment 2 lchai 2018-12-06 06:04:12 UTC
I also encountered the same issue, after change the CDROM device, the node name also changed.

Before:
cdrom1: json:{"throttle-group": "foo", "driver": "throttle", "file": {"driver": "file", "filename": "/home/kvmtest/cd1"}} (throttle, read-only)
    Attached to:      cd1
    Removable device: not locked, tray open
    Cache mode:       writeback, ignore flushes

Change the device:
{"execute":"blockdev-change-medium","arguments":{"id":"cd1","filename":"/home/kvmtest/cd2"}}

After:
#block329: /home/kvmtest/cd2 (raw, read-only)
    Attached to:      cd1
    Removable device: locked, tray closed
    Cache mode:       writeback, ignore flushes


Test environment info:
Host:
kernel-4.18.0-40.el8.x86_64
qemu-kvm-3.0.0-2.module+el8+2208+e41b12e0.x86_64

Comment 3 Xueqiang Wei 2018-12-11 10:15:37 UTC
According to https://bugzilla.redhat.com/show_bug.cgi?id=1655972#c4, it's fixed in upstream 3.1.0-rc5.


I retested on QEMU 3.1.0 rc5, not hit this issue. Migrate successfully and guest works well. 

kernel-4.18.0-50.el8.x86_64
QEMU 3.1.0 rc5 


kwolf,

If need to set bug status to ON_QA or CURRENTRELEASE ? 

Thanks.

Comment 5 Kevin Wolf 2018-12-14 09:19:23 UTC
I think we're still waiting for the first proper fast train build, so I'm moving it to POST for now. As soon as the 3.1.0-1 build is there, it can move to MODIFIED and ON_QA.

Comment 10 Xueqiang Wei 2019-01-29 11:15:02 UTC
According to Comment 8, set bug status to VERIFIED.

Comment 12 errata-xmlrpc 2019-05-29 16:04:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1293


Note You need to log in before you can comment on or make changes to this bug.