Bug 1378816

Summary: Core dump when use "data-plane" and execute change cd
Product: Red Hat Enterprise Linux 7 Reporter: yduan
Component: qemu-kvm-rhevAssignee: Fam Zheng <famz>
Status: CLOSED ERRATA QA Contact: aihua liang <aliang>
Severity: high Docs Contact:
Priority: urgent    
Version: 7.3CC: chayang, coli, famz, jsnow, juzhang, lijin, lmiksik, michen, mtessun, qzhang, stefanha, virt-maint, xfu, yhong
Target Milestone: rcKeywords: Regression, ZStream
Target Release: 7.4   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-12.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1461837 (view as bug list) Environment:
Last Closed: 2017-08-01 23:37:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1401400, 1461837    

Description yduan 2016-09-23 10:20:02 UTC 
Description of problem:
Core dump when change cd

Version-Release number of selected component (if applicable):
kernel: 3.10.0-510.el7.x86_64
qemu-kvm-rhev-2.6.0-26.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Start VM with following commands:
/usr/libexec/qemu-kvm \
 -device virtio-scsi-pci,id=scsi0 \
 -drive file=sysdisk.qcow2,format=qcow2,id=drive_sysdisk,if=none,cache=none,aio=native,werror=stop,rerror=stop \
 -device scsi-hd,drive=drive_sysdisk,bus=scsi0.0,id=device_sysdisk,bootindex=0 \
 -object iothread,id=iothread0 \
 -device virtio-scsi-pci,id=scsi1,iothread=iothread0 \
 -drive file=/home/test/backup/cd.iso,if=none,cache=none,media=cdrom,id=drive_cd,readonly=on \
 -device scsi-cd,bus=scsi1.0,drive=drive_cd,id=device_cd \

2.Eject cdrom:
(qemu) info block
drive_sysdisk (#block101): sysdisk.qcow2 (qcow2)
    Cache mode:       writeback, direct
drive_cd (#block367): /home/test/backup/cd.iso (raw, read-only)
    Removable device: locked, tray closed
    Cache mode:       writeback, direct

(qemu) eject drive_cd
Device 'drive_cd' is locked and force was not specified, wait for tray to open and try again

(qemu) info block
drive_sysdisk (#block101): sysdisk.qcow2 (qcow2)
    Cache mode:       writeback, direct
drive_cd (#block367): /home/test/backup/cd.iso (raw, read-only)
    Removable device: not locked, tray open
    Cache mode:       writeback, direct

3.Change cd:
(qemu) change drive_cd /home/test/backup/cd.iso
(qemu) qemu-kvm: /builddir/build/BUILD/qemu-2.6.0/hw/scsi/virtio-scsi.c:242: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.
eject.sh: line 40: 23022 Aborted                 (core dumped)

Actual results:
Core dump

Expected results:
Change cd successfully.

Additional info:
1.Not reproducible with qemu-kvm-rhev-2.3.0-31.el7_2.22.
(qemu) eject drive_cd
Device 'drive_cd' is busy: block device is in use by data plane
(qemu) change drive_cd /home/test/backup/cd.iso
Device 'drive_cd' is busy: block device is in use by data plane

2.Not reproducible without "data-plane".
(qemu) info block
drive_sysdisk (#block194): sysdisk.qcow2 (qcow2)
    Cache mode:       writeback, direct
drive_cd (#block326): /home/test/backup/cd.iso (raw, read-only)
    Removable device: locked, tray closed
    Cache mode:       writeback, direct
(qemu) eject drive_cd
Device 'drive_cd' is locked and force was not specified, wait for tray to open and try again
(qemu) info block
drive_sysdisk (#block194): sysdisk.qcow2 (qcow2)
    Cache mode:       writeback, direct
drive_cd (#block326): /home/test/backup/cd.iso (raw, read-only)
    Removable device: not locked, tray open
    Cache mode:       writeback, direct
(qemu) change drive_cd /home/test/backup/cd.iso
(qemu) info block
drive_sysdisk (#block194): sysdisk.qcow2 (qcow2)
    Cache mode:       writeback, direct
drive_cd (#block504): /home/test/backup/cd.iso (raw, read-only)
    Removable device: locked, tray closed
    Cache mode:       writeback, direct

3.(gdb) bt
#0  0x00007fffed95f1d7 in raise () from /lib64/libc.so.6
#1  0x00007fffed9608c8 in abort () from /lib64/libc.so.6
#2  0x00007fffed958146 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007fffed9581f2 in __assert_fail () from /lib64/libc.so.6
#4  0x0000555555733fff in virtio_scsi_ctx_check (s=<optimized out>, s=<optimized out>, d=0x555557a0ed80)
    at /usr/src/debug/qemu-2.6.0/hw/scsi/virtio-scsi.c:242
#5  0x00005555557aa52e in virtio_scsi_ctx_check (s=<optimized out>, s=<optimized out>, d=0x555557a0ed80)
    at /usr/src/debug/qemu-2.6.0/hw/scsi/virtio-scsi.c:242
#6  virtio_scsi_handle_cmd_req_prepare (req=0x55555804b180, s=0x5555596e8340)
    at /usr/src/debug/qemu-2.6.0/hw/scsi/virtio-scsi.c:547
#7  virtio_scsi_handle_cmd_vq (s=0x5555596e8340, vq=0x555558b440f0)
    at /usr/src/debug/qemu-2.6.0/hw/scsi/virtio-scsi.c:580
#8  0x000055555596d572 in aio_dispatch (ctx=ctx@entry=0x555556ce3c80) at aio-posix.c:330
#9  0x000055555596d788 in aio_poll (ctx=0x555556ce3c80, blocking=<optimized out>) at aio-posix.c:479
#10 0x0000555555837ad9 in iothread_run (opaque=0x555556cca640) at iothread.c:46
#11 0x00007fffedcf2dc5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fffeda2173d in clone () from /lib64/libc.so.6

(gdb) bt full
#0  0x00007fffed95f1d7 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007fffed9608c8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007fffed958146 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007fffed9581f2 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4  0x0000555555733fff in virtio_scsi_ctx_check (s=<optimized out>, s=<optimized out>, d=0x555557a0ed80)
    at /usr/src/debug/qemu-2.6.0/hw/scsi/virtio-scsi.c:242
No locals.
#5  0x00005555557aa52e in virtio_scsi_ctx_check (s=<optimized out>, s=<optimized out>, d=0x555557a0ed80)
    at /usr/src/debug/qemu-2.6.0/hw/scsi/virtio-scsi.c:242
No locals.
#6  virtio_scsi_handle_cmd_req_prepare (req=0x55555804b180, s=0x5555596e8340)
    at /usr/src/debug/qemu-2.6.0/hw/scsi/virtio-scsi.c:547
        vs = 0x5555596e8340
        rc = <optimized out>
#7  virtio_scsi_handle_cmd_vq (s=0x5555596e8340, vq=0x555558b440f0)
    at /usr/src/debug/qemu-2.6.0/hw/scsi/virtio-scsi.c:580
        req = 0x55555804b180
---Type <return> to continue, or q <return> to quit--- 
        next = <optimized out>
        reqs = {tqh_first = 0x0, tqh_last = 0x7fffe3bbf8c0}
#8  0x000055555596d572 in aio_dispatch (ctx=ctx@entry=0x555556ce3c80) at aio-posix.c:330
        tmp = <optimized out>
        revents = <optimized out>
        node = 0x55555817c0c0
        progress = false
#9  0x000055555596d788 in aio_poll (ctx=0x555556ce3c80, blocking=<optimized out>) at aio-posix.c:479
        node = <optimized out>
        i = <optimized out>
        ret = 1
        progress = false
        timeout = <optimized out>
        __PRETTY_FUNCTION__ = "aio_poll"
#10 0x0000555555837ad9 in iothread_run (opaque=0x555556cca640) at iothread.c:46
        iothread = 0x555556cca640
        blocking = <optimized out>
#11 0x00007fffedcf2dc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#12 0x00007fffeda2173d in clone () from /lib64/libc.so.6
Comment 1 Fam Zheng 2016-09-23 11:23:26 UTC 
A possible fix:

https://lists.nongnu.org/archive/html/qemu-devel/2016-09/msg06185.html
Comment 4 yduan 2016-09-23 12:19:53 UTC 
This issue is not reproducible with qemu-kvm-rhev-2.6.0-9.el7.x86_64 and reproducible with qemu-kvm-rhev-2.6.0-10.el7.x86_64.
Comment 5 Ademar Reis 2016-09-23 17:35:58 UTC 
(In reply to yduan from comment #4)
> This issue is not reproducible with qemu-kvm-rhev-2.6.0-9.el7.x86_64 and
> reproducible with qemu-kvm-rhev-2.6.0-10.el7.x86_64.

Fam, can you please bisect it? John, Stefan: you're welcome to take a look as well.

I'm removing the blocker flag, leaving this as a candidate to the z-stream.
Comment 7 Fam Zheng 2016-09-24 03:50:23 UTC 
(In reply to Ademar Reis from comment #5)
> Fam, can you please bisect it? John, Stefan: you're welcome to take a look
> as well.

The bad commit is removal of the op blocker from virtio-scsi dataplane, backported in qemu-kvm-rhev-2.6.0-10.el7.x86_64:

commit e6e5df6e5c7a87cdc0471c9bc60a1946102d4274
Author: Fam Zheng <famz>
Date:   Wed Jun 22 02:20:38 2016 +0200

    virtio-scsi: Remove op blocker for dataplane

It applies to upstream as well. The fix was posted as said in commit 1.
Comment 8 Ademar Reis 2017-04-10 20:26:02 UTC 
(In reply to Fam Zheng from comment #7)
> (In reply to Ademar Reis from comment #5)
> > Fam, can you please bisect it? John, Stefan: you're welcome to take a look
> > as well.
> 
> The bad commit is removal of the op blocker from virtio-scsi dataplane,
> backported in qemu-kvm-rhev-2.6.0-10.el7.x86_64:
> 
> commit e6e5df6e5c7a87cdc0471c9bc60a1946102d4274
> Author: Fam Zheng <famz>
> Date:   Wed Jun 22 02:20:38 2016 +0200
> 
>     virtio-scsi: Remove op blocker for dataplane
> 
> It applies to upstream as well. The fix was posted as said in commit 1.

So I'm confused now... Do you mean the fix is already upstream, in QEMU-2.9? If that's the case, this BZ should be POST (fixed-in-version: qemu-2.9).
Comment 12 Fam Zheng 2017-04-11 16:46:37 UTC 
The patch was proposed, and denied after discussion. So no, not in 2.9.. We'll have to backport it from 2.10.
Comment 20 Miroslav Rezanina 2017-06-20 06:02:03 UTC 
Fix included in qemu-kvm-rhev-2.9.0-12.el7
Comment 22 aihua liang 2017-06-20 08:06:54 UTC 
Has verified, scsi-cd is not supported by data plane.

Test version:
  kernel: 3.10.0-682.el7.x86_64
  qemu-kvm-rhev:qemu-kvm-rhev-2.9.0-12.el7.x86_64

Test Steps:
 1.Start guest with qemu cmd:
     /usr/libexec/qemu-kvm \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20170614-233639-etu9X2zc,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20170614-233639-etu9X2zc,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idhq2DAN  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20170614-233639-etu9X2zc,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20170614-233639-etu9X2zc,path=/var/tmp/seabios-20170614-233639-etu9X2zc,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20170614-233639-etu9X2zc,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -object iothread,id=iothread0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel74-64-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=1,bus=pci.0,addr=0x3 \
    -drive id=data,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/data_disk.img \
    -device virtio-blk-pci,id=data1,drive=data,bus=pci.0 \
    -device virtio-scsi-pci,id=scsi1,iothread=iothread0 \
    -drive if=none,cache=none,media=cdrom,id=drive_cd,readonly=on,file=/home/kvm_autotest_root/iso/linux/RHEL7.4-Server-x86_64.iso \
    -device scsi-cd,id=device_cd,drive=drive_cd,bus=scsi1.0 \
    -device virtio-net-pci,mac=9a:43:44:45:46:47,id=idvMp6XX,vectors=4,netdev=id9qJxPT,bus=pci.0 \
    -netdev tap,id=id9qJxPT,vhost=on \
    -m 4096  \
    -smp 6,cores=2,threads=1,sockets=3  \
    -cpu host \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off  \
    -no-shutdown \
    -enable-kvm \
    -monitor stdio \
    -spice ipv4,port=5000,disable-ticketing \

Test Result:
 Start guest failed with error msg: 
  qemu) qemu-kvm: -device scsi-cd,id=device_cd,drive=drive_cd,bus=scsi1.0: scsi-cd is not supported by data plane
Comment 24 errata-xmlrpc 2017-08-01 23:37:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 25 errata-xmlrpc 2017-08-02 01:14:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 26 errata-xmlrpc 2017-08-02 02:06:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 27 errata-xmlrpc 2017-08-02 02:47:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 28 errata-xmlrpc 2017-08-02 03:12:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 29 errata-xmlrpc 2017-08-02 03:32:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392