Bug 1599515 - qemu core-dump with aio_read via hmp (util/qemu-thread-posix.c:64: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed)
Summary: qemu core-dump with aio_read via hmp (util/qemu-thread-posix.c:64: qemu_mutex...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-10 01:46 UTC by Qianqian Zhu
Modified: 2018-11-01 11:15 UTC (History)
6 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-8.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-01 11:13:00 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3443 None None None 2018-11-01 11:15:19 UTC

Description Qianqian Zhu 2018-07-10 01:46:13 UTC
Description of problem:
Perform qemu-io aio_read in hmp will make qemu core dump

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.12.0-7.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.# cat blkdebug.conf 
[inject-error]
event = "read_aio"
errno = "28"

2.# /usr/libexec/qemu-kvm -enable-kvm \
-m 2G \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4   \
-blockdev driver=raw,cache.direct=on,node-name=file_image1,file.filename=/home/rhel76-64-virtio-scsi.raw,file.driver=file,file.aio=native \
-blockdev driver=blkdebug,node-name=blockdev_image1,image=file_image1,config=blkdebug.conf  \
-device usb-storage,id=image1,drive=blockdev_image1,share-rw=on,rerror=stop,werror=stop \
-monitor stdio \
-usb

3.(qemu) qemu-io blockdev_image1 "aio_read -v 512 1k 1k"

Actual results:
(qemu) qemu-kvm: util/qemu-thread-posix.c:64: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed.
Aborted (core dumped)

(gdb) bt full
#0  0x00007fd615743207 in raise () at /lib64/libc.so.6
#1  0x00007fd6157448f8 in abort () at /lib64/libc.so.6
#2  0x00007fd61573c026 in __assert_fail_base () at /lib64/libc.so.6
#3  0x00007fd61573c0d2 in  () at /lib64/libc.so.6
#4  0x000055d516bcc0b2 in qemu_mutex_lock_impl (mutex=mutex@entry=0x55d519337468, file=file@entry=0x55d516d4b284 "block/accounting.c", line=line@entry=198) at util/qemu-thread-posix.c:64
        err = <optimized out>
        __PRETTY_FUNCTION__ = "qemu_mutex_lock_impl"
        __func__ = "qemu_mutex_lock_impl"
#5  0x000055d516b543cb in block_account_one_io (stats=0x55d519337468, cookie=cookie@entry=0x55d51acf8658, failed=failed@entry=false) at block/accounting.c:198
        s = <optimized out>
        time_ns = 623031128066611
        latency_ns = 8087040
        __PRETTY_FUNCTION__ = "block_account_one_io"
#6  0x000055d516b54867 in block_acct_done (stats=<optimized out>, cookie=cookie@entry=0x55d51acf8658) at block/accounting.c:224
#7  0x000055d516afd4e6 in aio_read_done (opaque=0x55d51acf8620, ret=<optimized out>) at qemu-io-cmds.c:1302
        ctx = 0x55d51acf8620
        t2 = {tv_sec = 1531120657, tv_usec = 447542}
#8  0x000055d516b36276 in blk_aio_complete (acb=0x55d51921f5e0) at block/block-backend.c:1331
#9  0x000055d516bde1aa in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at util/coroutine-ucontext.c:116
        self = 0x55d51a539b80
        co = 0x55d51a539b80
#10 0x00007fd615755010 in __start_context () at /lib64/libc.so.6
#11 0x00007ffd104e2190 in  ()
#12 0x0000000000000000 in  ()

Expected results:
No core dump, aio_read succeed.

Additional info:

Comment 3 Qianqian Zhu 2018-07-10 02:59:46 UTC
Reproduced with scsi-hd on qemu-kvm-rhev-2.10.0-16.el7.x86_64:
/usr/libexec/qemu-kvm \
-enable-kvm \
-m 2G \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4   \
-blockdev driver=raw,cache.direct=on,node-name=file_image1,file.filename=/home/rhel75-64-virtio-scsi.raw,file.driver=file,file.aio=native \
-blockdev driver=blkdebug,node-name=blockdev_image1,image=file_image1,config=blkdebug.conf \
-device scsi-hd,id=image1,drive=blockdev_image1,share-rw=on \
-monitor stdio

So, it is not usb related, and not regression.

Comment 4 Gu Nini 2018-07-10 03:11:11 UTC
It also can be reproduced with virtio-blk-pci.

And checked with Qianqian, the bug could only be reproduced with 'aio_read', while 'aio_write', 'read', 'write' without the issue.

If use the traditional way to specify a device as follows, even 'aio_read' could not induce the bug:
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/$1 \
    -device scsi-hd,drive=drive_image1,id=image1 \

Comment 5 Qianqian Zhu 2018-07-10 03:29:40 UTC
Tried removing blkdebug object, it is also reproducible. So the simplest reproducer here below:
1. Launch guest:
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4   \
-blockdev driver=raw,cache.direct=on,node-name=file_image1,file.filename=/home/rhel76-64-virtio-scsi.raw,file.driver=file,file.aio=native \
-device scsi-hd,id=image1,drive=file_image1,share-rw=on,rerror=stop,werror=stop \

2. (qemu) qemu-io file_image1 "aio_read -v 512 1k 1k"
(qemu) qemu-kvm: util/qemu-thread-posix.c:64: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed.
Aborted (core dumped)

Comment 7 Miroslav Rezanina 2018-07-24 14:42:59 UTC
Fix included in qemu-kvm-rhev-2.12.0-8.el7

Comment 9 aihua liang 2018-07-26 07:16:45 UTC
Verified in:
   kernel:3.10.0-918.el7.x86_64
   qemu-kvm-rhev: qemu-kvm-rhev-2.12.0-8.el7.x86_64
   test on both virtio_blk and virtio_scsi

The problem has been resolved, so set its status to "Verified".

Comment 10 errata-xmlrpc 2018-11-01 11:13:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443


Note You need to log in before you can comment on or make changes to this bug.