Bug 1086502

Summary: QEMU core dumped when blockdev_add with 'aio': 'native' but without 'cache' specified
Product: Red Hat Enterprise Linux 7 Reporter: Sibiao Luo <sluo>
Component: qemu-kvm-rhevAssignee: Kevin Wolf <kwolf>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: chayang, famz, flang, hhuang, juzhang, kwolf, michen, qzhang, rbalakri, rmainz, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-2.1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 09:45:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sibiao Luo 2014-04-11 03:15:09 UTC 
Description of problem:
QEMU core dumped when blockdev_add with 'aio': 'native' specified, but 'aio': 'threads' did not hit such issue.

Version-Release number of selected component (if applicable):
host info:
# uname -r && rpm -q qemu-kvm
3.10.0-121.el7.x86_64
qemu-kvm-1.5.3-60.el7.x86_64
guest info:
# uname -r
3.10.0-121.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.launch a QEMU guest.
2.hut-plug using blockdev_add with 'aio': 'native' specified.
{ "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio': 'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}}} }
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"disk1","id":"disk1","bus":"pci.0","scsi":"off","addr":"0x8"}}

Actual results:
after step 2, QEMU Segmentation fault (core dumped), i will paste the full bt log later.
Core was generated by `/usr/libexec/qemu-kvm -M pc -S -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f2369f0670c in qmp_blockdev_add (options=0x7f236d4457d0, errp=errp@entry=0x7fff8f935c68) at blockdev.c:1771
1771	        bool direct = options->cache->has_direct && options->cache->direct;

(gdb) bt
#0  0x00007f2369f0670c in qmp_blockdev_add (options=0x7f236d4457d0, errp=errp@entry=0x7fff8f935c68) at blockdev.c:1771
#1  0x00007f2369fce945 in qmp_marshal_input_blockdev_add (mon=<optimized out>, qdict=<optimized out>, 
    ret=<optimized out>) at qmp-marshal.c:3543
#2  0x00007f236a055e57 in qmp_call_cmd (cmd=<optimized out>, params=0x7f236d677a70, mon=0x7f236c831670)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4509
#3  handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4575
#4  0x00007f236a102582 in json_message_process_token (lexer=0x7f236c831700, token=0x7f236d4452f0, type=JSON_OPERATOR, 
    x=172, y=3) at qobject/json-streamer.c:87
#5  0x00007f236a111b0f in json_lexer_feed_char (lexer=lexer@entry=0x7f236c831700, ch=<optimized out>, 
    flush=flush@entry=false) at qobject/json-lexer.c:303
#6  0x00007f236a111bde in json_lexer_feed (lexer=0x7f236c831700, buffer=<optimized out>, size=<optimized out>)
    at qobject/json-lexer.c:356
#7  0x00007f236a102719 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>)
    at qobject/json-streamer.c:110
#8  0x00007f236a054ba3 in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4596
#9  0x00007f2369fc3d31 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff8f935e70 "}>\200l#\177", s=0x7f236c804a20)
    at qemu-char.c:167
#10 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f236c804a20) at qemu-char.c:2491
#11 0x00007f23692fdac6 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#12 0x00007f2369f96aaa in glib_pollfds_poll () at main-loop.c:187
#13 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232
#14 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464
#15 0x00007f2369ebce50 in main_loop () at vl.c:1988
#16 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4360
(gdb) 

Expected results:
It should no any qemu core dumped occur.

Additional info:
# /usr/libexec/qemu-kvm -M pc -S -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-7.0-20140409.0_Server_x86_64.qcow2,if=none,id=drive-ide-disk,format=qcow2,cache=none,aio=native,discard=on -device ide-hd,bus=ide.0,unit=0,drive=drive-ide-disk,id=ide-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:B6:40:21,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice disable-ticketing,port=5931 -monitor stdio
Comment 1 Sibiao Luo 2014-04-11 03:16:11 UTC 
(gdb) bt full
#0  0x00007f2369f0670c in qmp_blockdev_add (options=0x7f236d4457d0, errp=errp@entry=0x7fff8f935c68) at blockdev.c:1771
        direct = <optimized out>
        ov = 0x7f236d466380
        obj = <optimized out>
        qdict = <optimized out>
        local_err = 0x0
#1  0x00007f2369fce945 in qmp_marshal_input_blockdev_add (mon=<optimized out>, qdict=<optimized out>, 
    ret=<optimized out>) at qmp-marshal.c:3543
        local_err = 0x0
        errp = 0x7fff8f935c68
        args = <optimized out>
        mi = 0x7f236e100ba0
        md = <optimized out>
        v = <optimized out>
        options = 0x7f236d4457d0
#2  0x00007f236a055e57 in qmp_call_cmd (cmd=<optimized out>, params=0x7f236d677a70, mon=0x7f236c831670)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4509
        ret = <optimized out>
        data = 0x0
#3  handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4575
        err = <optimized out>
        obj = <optimized out>
        input = <optimized out>
        args = 0x7f236d677a70
        cmd_name = <optimized out>
        mon = 0x7f236c831670
#4  0x00007f236a102582 in json_message_process_token (lexer=0x7f236c831700, token=0x7f236d4452f0, type=JSON_OPERATOR, 
    x=172, y=3) at qobject/json-streamer.c:87
        parser = 0x7f236c8316f8
        dict = 0x7f236d675a30
#5  0x00007f236a111b0f in json_lexer_feed_char (lexer=lexer@entry=0x7f236c831700, ch=<optimized out>, 
    flush=flush@entry=false) at qobject/json-lexer.c:303
        new_state = 100
#6  0x00007f236a111bde in json_lexer_feed (lexer=0x7f236c831700, buffer=<optimized out>, size=<optimized out>)
    at qobject/json-lexer.c:356
        err = <optimized out>
        i = <optimized out>
#7  0x00007f236a102719 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>)
    at qobject/json-streamer.c:110
No locals.
#8  0x00007f236a054ba3 in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4596
        old_mon = 0x0
#9  0x00007f2369fc3d31 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff8f935e70 "}>\200l#\177", s=0x7f236c804a20)
    at qemu-char.c:167
No locals.
#10 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f236c804a20) at qemu-char.c:2491
        chr = 0x7f236c804a20
        s = 0x7f236c804af0
        buf = "}>\200l#\177\000\000k;\374i#\177\000\000\227\005\022\005A\a\236\003\001\000\000\000\000\000\000\000\r\000\000\000\000\000\000\000\340\252\016n#\177\000\000\360\250\016n#\177\000\000\360\250\016n#\177\000\000\360\250\016n#\177\000\000\000\221\267\347\357>\352R \346Ym#\177\000\000`_\223\217\377\177\000\000\360\250\016n#\177\000\000`;\\m#\177\000\000\2---Type <return> to continue, or q <return> to quit---
00\025\241l#\177\000\000\021\"\020j#\177\000\000#\000\000\000\000\000\000\000\200\321\320l#\177\000\000`_\223\217\377\177\000\000\200!\020j#\177\000\000\060\n\000\000\000\000\000\000\345\023\020j#\177\000\000 \346Ym#\177\000\000c\v\020j#\177\000\000 \346Ym#\177\000\000"...
        len = <optimized out>
        size = <optimized out>
#11 0x00007f23692fdac6 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
No symbol table info available.
#12 0x00007f2369f96aaa in glib_pollfds_poll () at main-loop.c:187
        context = 0x7f236c803e00
        pfds = <optimized out>
#13 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232
        ret = 2
        spin_counter = 0
#14 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464
        ret = 2
        timeout = 4294967295
#15 0x00007f2369ebce50 in main_loop () at vl.c:1988
        nonblocking = <optimized out>
        last_io = 2
#16 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4360
        i = <optimized out>
        snapshot = 0
        linux_boot = 0
        icount_option = 0x0
        initrd_filename = 0x0
        kernel_filename = 0x0
        kernel_cmdline = 0x7f236a16a8e0 ""
        boot_order = 0x7f236a122806 "cad"
        ds = <optimized out>
        cyls = 0
        heads = 0
        secs = 0
        translation = 0
        hda_opts = <optimized out>
        opts = 0x7f236c8021f0
        machine_opts = <optimized out>
        olist = <optimized out>
        optind = 59
        optarg = 0x7fff8f9397e7 "stdio"
        loadvm = 0x0
        machine = 0x7f236a4f63c0 <pc_machine_rhel700>
        cpu_model = 0x7fff8f939306 "host"
        vga_model = 0x7f236a14d4df "cirrus"
        pid_file = 0x0
        incoming = 0x0
        show_vnc_port = 0
        defconfig = <optimized out>
        userconfig = 6
        log_mask = <optimized out>
        log_file = 0x0
        mem_trace = {malloc = 0x7f2369ffd720 <malloc_and_trace>, realloc = 0x7f2369ffd700 <realloc_and_trace>, 
          free = 0x7f2369ffd6f0 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
        trace_events = 0x0
        trace_file = 0x0
        __PRETTY_FUNCTION__ = "main"
        args = {machine = 0x7f236a4f63c0 <pc_machine_rhel700>, ram_size = 2147483648, 
          boot_device = 0x7f236a122806 "cad", kernel_filename = 0x0, kernel_cmdline = 0x7f236a16a8e0 "", 
          initrd_filename = 0x0, cpu_model = 0x7fff8f939306 "host"}
(gdb)
Comment 2 Sibiao Luo 2014-04-11 03:23:52 UTC 
The orignial QMP command line(__com.redhat_drive_add/device_add) did not hit this issue.
e.g:
{"execute":"__com.redhat_drive_add","arguments": {"file":"/home/my-data-disk.raw","format":"raw","id":"drive-data-disk","aio":"native","werror":"stop","rerror":"stop"}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"drive-data-disk","id":"data-disk","bus":"pci.0","scsi":"off","addr":"0x9"}}
{"return": {}}

Best Regards,
sluo
Comment 3 Sibiao Luo 2014-04-11 05:58:02 UTC 
If append the 'cache' specifying ('writeback': true|false) which did not hit this issue.
- e.g.1, 'writeback': *false*.
{ "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio':'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}, 'cache': { 'writeback': false, 'direct': true, 'no-flush': false }}} }
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"disk1","id":"disk1","bus":"pci.0","scsi":"off","addr":"0x8"}}
{"return": {}}

- e.g.1, 'writeback': *true*.
{ "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio':'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}, 'cache': { 'writeback': true, 'direct': true, 'no-flush': false }}} }
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"disk1","id":"disk1","bus":"pci.0","scsi":"off","addr":"0x8"}}
{"return": {}}

Best Regards,
sluo
Comment 4 Kevin Wolf 2014-04-11 13:14:21 UTC 
Fixed in upstream commit c6e0bd9b. We'll get it with the 7.1 rebase.
Comment 6 juzhang 2014-07-28 01:30:59 UTC 
Since this bz will be fixed in qemu2.1, I guess we should update the component to qemu-kvm-rhev.

Best Regards,
Junyi
Comment 9 langfang 2014-08-28 05:07:22 UTC 
Reproduce this bug as follow version:
Host

# uname -r
3.10.0-144.el7.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-1.5.3-60.el7_0_0.6.x86_64

Guest:rhel6.6

1.Boot guest

2.remote-viewer spice://10.66.7.55:5931
3.hut-plug using blockdev_add with 'aio': 'native' specified.
{ "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio': 'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}}} }
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"disk1","id":"disk1","bus":"pci.0","scsi":"off","addr":"0x8"}}

Results: Segmentation fault

Program received signal SIGSEGV, Segmentation fault.
0x000055555565128c in qmp_blockdev_add (options=0x555556d853b0, 
    errp=errp@entry=0x7fffffffca48) at blockdev.c:1771
1771	        bool direct = options->cache->has_direct && options->cache->direct;
Missing separate debuginfos, use: debuginfo-install alsa-lib-1.0.27.2-3.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 cyrus-sasl-lib-2.1.26-17.el7.x86_64 cyrus-sasl-md5-2.1.26-17.el7.x86_64 cyrus-sasl-plain-2.1.26-17.el7.x86_64 dbus-libs-1.6.12-8.el7.x86_64 flac-libs-1.3.0-4.el7.x86_64 glib2-2.36.3-5.el7.x86_64 glibc-2.17-55.el7.x86_64 glusterfs-api-3.4.0.59rhs-1.el7.x86_64 glusterfs-libs-3.4.0.59rhs-1.el7.x86_64 gmp-5.1.1-5.el7.x86_64 gnutls-3.1.18-8.el7.x86_64 gsm-1.0.13-11.el7.x86_64 json-c-0.11-3.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.11.3-49.el7.x86_64 libICE-1.0.8-7.el7.x86_64 libSM-1.2.1-7.el7.x86_64 libX11-1.6.0-2.1.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXext-1.3.2-2.1.el7.x86_64 libXi-1.7.2-2.1.el7.x86_64 libXtst-1.2.2-2.1.el7.x86_64 libaio-0.3.109-12.el7.x86_64 libasyncns-0.8-7.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-4.el7.x86_64 libdb-5.3.21-17.el7.x86_64 libgcc-4.8.2-16.el7.x86_64 libgcrypt-1.5.3-4.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibverbs-1.1.7-6.el7.x86_64 libiscsi-1.9.0-6.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libnl-1.1.4-3.el7.x86_64 libogg-1.3.0-7.el7.x86_64 libpng-1.5.13-5.el7.x86_64 librdmacm-1.0.17.1-1.el7.x86_64 libseccomp-2.1.1-2.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libsndfile-1.0.25-9.el7.x86_64 libtasn1-3.3-3.el7.x86_64 libusbx-1.0.15-4.el7.x86_64 libuuid-2.23.2-16.el7.x86_64 libvorbis-1.3.3-8.el7.x86_64 libxcb-1.9-5.el7.x86_64 nettle-2.7.1-2.el7.x86_64 nspr-4.10.2-4.el7.x86_64 nss-3.15.4-6.el7.x86_64 nss-softokn-freebl-3.15.4-2.el7.x86_64 nss-util-3.15.4-2.el7.x86_64 openssl-libs-1.0.1e-34.el7.x86_64 p11-kit-0.18.7-4.el7.x86_64 pcre-8.32-12.el7.x86_64 pixman-0.32.4-3.el7.x86_64 pulseaudio-libs-3.0-22.el7.x86_64 tcp_wrappers-libs-7.6-77.el7.x86_64 usbredir-0.6-7.el7.x86_64 xz-libs-5.1.2-8alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) brt
Undefined command: "brt".  Try "help".
(gdb) bt
#0  0x000055555565128c in qmp_blockdev_add (options=0x555556d853b0, 
    errp=errp@entry=0x7fffffffca48) at blockdev.c:1771
#1  0x000055555571b225 in qmp_marshal_input_blockdev_add (
    mon=<optimized out>, qdict=<optimized out>, ret=<optimized out>)
    at qmp-marshal.c:3893
#2  0x00005555557a2c47 in qmp_call_cmd (cmd=<optimized out>, 
    params=0x55555682a870, mon=0x555556515930)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4509
#3  handle_qmp_command (parser=<optimized out>, tokens=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4575
#4  0x000055555584f7a2 in json_message_process_token (
    lexer=0x5555564f7330, token=0x555556d850d0, type=JSON_OPERATOR, 
    x=172, y=3) at qobject/json-streamer.c:87
#5  0x000055555585ed2f in json_lexer_feed_char (
    lexer=lexer@entry=0x5555564f7330, ch=<optimized out>, 
    flush=flush@entry=false) at qobject/json-lexer.c:303
#6  0x000055555585edfe in json_lexer_feed (lexer=0x5555564f7330, 
    buffer=<optimized out>, size=<optimized out>)
    at qobject/json-lexer.c:356
#7  0x000055555584f939 in json_message_parser_feed (
    parser=<optimized out>, buffer=<optimized out>, 
    size=<optimized out>) at qobject/json-streamer.c:110
#8  0x00005555557a1993 in monitor_control_read (opaque=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    buf=<optimized out>, size=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4596
#9  0x000055555570f541 in qemu_chr_be_write (len=<optimized out>, 
    buf=0x7fffffffcc50 "}\210NVUU", s=0x5555564e8d70) at qemu-char.c:167
#10 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, 
    opaque=0x5555564e8d70) at qemu-char.c:2492
#11 0x00007ffff74edac6 in g_main_context_dispatch ()
   from /lib64/libglib-2.0.so.0
#12 0x00005555556e1a4a in glib_pollfds_poll () at main-loop.c:187
#13 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232
#14 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464
#15 0x0000555555602f50 in main_loop () at vl.c:1988
#16 main (argc=<optimized out>, argv=<optimized out>, 
    envp=<optimized out>) at vl.c:4359

Verify this bug as follow version:
Host:
# uname -r
3.10.0-144.el7.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-2.1.0-2.el7.x86_64

Guest:rhel7

Steps as same as reproduce

Results:

{ "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio': 'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}}} }
{"error": {"class": "GenericError", "desc": "aio=native requires cache.direct=true"}}

Forbit add block without cache paramter if with aio.


According to above test ,this bug has been fixed.
Comment 12 errata-xmlrpc 2015-03-05 09:45:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0624.html