Hide Forgot
Description of problem: QEMU core dumped when blockdev_add with 'aio': 'native' specified, but 'aio': 'threads' did not hit such issue. Version-Release number of selected component (if applicable): host info: # uname -r && rpm -q qemu-kvm 3.10.0-121.el7.x86_64 qemu-kvm-1.5.3-60.el7.x86_64 guest info: # uname -r 3.10.0-121.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.launch a QEMU guest. 2.hut-plug using blockdev_add with 'aio': 'native' specified. { "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio': 'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}}} } {"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"disk1","id":"disk1","bus":"pci.0","scsi":"off","addr":"0x8"}} Actual results: after step 2, QEMU Segmentation fault (core dumped), i will paste the full bt log later. Core was generated by `/usr/libexec/qemu-kvm -M pc -S -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,c'. Program terminated with signal 11, Segmentation fault. #0 0x00007f2369f0670c in qmp_blockdev_add (options=0x7f236d4457d0, errp=errp@entry=0x7fff8f935c68) at blockdev.c:1771 1771 bool direct = options->cache->has_direct && options->cache->direct; (gdb) bt #0 0x00007f2369f0670c in qmp_blockdev_add (options=0x7f236d4457d0, errp=errp@entry=0x7fff8f935c68) at blockdev.c:1771 #1 0x00007f2369fce945 in qmp_marshal_input_blockdev_add (mon=<optimized out>, qdict=<optimized out>, ret=<optimized out>) at qmp-marshal.c:3543 #2 0x00007f236a055e57 in qmp_call_cmd (cmd=<optimized out>, params=0x7f236d677a70, mon=0x7f236c831670) at /usr/src/debug/qemu-1.5.3/monitor.c:4509 #3 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4575 #4 0x00007f236a102582 in json_message_process_token (lexer=0x7f236c831700, token=0x7f236d4452f0, type=JSON_OPERATOR, x=172, y=3) at qobject/json-streamer.c:87 #5 0x00007f236a111b0f in json_lexer_feed_char (lexer=lexer@entry=0x7f236c831700, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 #6 0x00007f236a111bde in json_lexer_feed (lexer=0x7f236c831700, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 #7 0x00007f236a102719 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 #8 0x00007f236a054ba3 in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4596 #9 0x00007f2369fc3d31 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff8f935e70 "}>\200l#\177", s=0x7f236c804a20) at qemu-char.c:167 #10 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f236c804a20) at qemu-char.c:2491 #11 0x00007f23692fdac6 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 #12 0x00007f2369f96aaa in glib_pollfds_poll () at main-loop.c:187 #13 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232 #14 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464 #15 0x00007f2369ebce50 in main_loop () at vl.c:1988 #16 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4360 (gdb) Expected results: It should no any qemu core dumped occur. Additional info: # /usr/libexec/qemu-kvm -M pc -S -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-7.0-20140409.0_Server_x86_64.qcow2,if=none,id=drive-ide-disk,format=qcow2,cache=none,aio=native,discard=on -device ide-hd,bus=ide.0,unit=0,drive=drive-ide-disk,id=ide-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:B6:40:21,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice disable-ticketing,port=5931 -monitor stdio
(gdb) bt full #0 0x00007f2369f0670c in qmp_blockdev_add (options=0x7f236d4457d0, errp=errp@entry=0x7fff8f935c68) at blockdev.c:1771 direct = <optimized out> ov = 0x7f236d466380 obj = <optimized out> qdict = <optimized out> local_err = 0x0 #1 0x00007f2369fce945 in qmp_marshal_input_blockdev_add (mon=<optimized out>, qdict=<optimized out>, ret=<optimized out>) at qmp-marshal.c:3543 local_err = 0x0 errp = 0x7fff8f935c68 args = <optimized out> mi = 0x7f236e100ba0 md = <optimized out> v = <optimized out> options = 0x7f236d4457d0 #2 0x00007f236a055e57 in qmp_call_cmd (cmd=<optimized out>, params=0x7f236d677a70, mon=0x7f236c831670) at /usr/src/debug/qemu-1.5.3/monitor.c:4509 ret = <optimized out> data = 0x0 #3 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4575 err = <optimized out> obj = <optimized out> input = <optimized out> args = 0x7f236d677a70 cmd_name = <optimized out> mon = 0x7f236c831670 #4 0x00007f236a102582 in json_message_process_token (lexer=0x7f236c831700, token=0x7f236d4452f0, type=JSON_OPERATOR, x=172, y=3) at qobject/json-streamer.c:87 parser = 0x7f236c8316f8 dict = 0x7f236d675a30 #5 0x00007f236a111b0f in json_lexer_feed_char (lexer=lexer@entry=0x7f236c831700, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 new_state = 100 #6 0x00007f236a111bde in json_lexer_feed (lexer=0x7f236c831700, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 err = <optimized out> i = <optimized out> #7 0x00007f236a102719 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals. #8 0x00007f236a054ba3 in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4596 old_mon = 0x0 #9 0x00007f2369fc3d31 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff8f935e70 "}>\200l#\177", s=0x7f236c804a20) at qemu-char.c:167 No locals. #10 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f236c804a20) at qemu-char.c:2491 chr = 0x7f236c804a20 s = 0x7f236c804af0 buf = "}>\200l#\177\000\000k;\374i#\177\000\000\227\005\022\005A\a\236\003\001\000\000\000\000\000\000\000\r\000\000\000\000\000\000\000\340\252\016n#\177\000\000\360\250\016n#\177\000\000\360\250\016n#\177\000\000\360\250\016n#\177\000\000\000\221\267\347\357>\352R \346Ym#\177\000\000`_\223\217\377\177\000\000\360\250\016n#\177\000\000`;\\m#\177\000\000\2---Type <return> to continue, or q <return> to quit--- 00\025\241l#\177\000\000\021\"\020j#\177\000\000#\000\000\000\000\000\000\000\200\321\320l#\177\000\000`_\223\217\377\177\000\000\200!\020j#\177\000\000\060\n\000\000\000\000\000\000\345\023\020j#\177\000\000 \346Ym#\177\000\000c\v\020j#\177\000\000 \346Ym#\177\000\000"... len = <optimized out> size = <optimized out> #11 0x00007f23692fdac6 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 No symbol table info available. #12 0x00007f2369f96aaa in glib_pollfds_poll () at main-loop.c:187 context = 0x7f236c803e00 pfds = <optimized out> #13 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232 ret = 2 spin_counter = 0 #14 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464 ret = 2 timeout = 4294967295 #15 0x00007f2369ebce50 in main_loop () at vl.c:1988 nonblocking = <optimized out> last_io = 2 #16 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4360 i = <optimized out> snapshot = 0 linux_boot = 0 icount_option = 0x0 initrd_filename = 0x0 kernel_filename = 0x0 kernel_cmdline = 0x7f236a16a8e0 "" boot_order = 0x7f236a122806 "cad" ds = <optimized out> cyls = 0 heads = 0 secs = 0 translation = 0 hda_opts = <optimized out> opts = 0x7f236c8021f0 machine_opts = <optimized out> olist = <optimized out> optind = 59 optarg = 0x7fff8f9397e7 "stdio" loadvm = 0x0 machine = 0x7f236a4f63c0 <pc_machine_rhel700> cpu_model = 0x7fff8f939306 "host" vga_model = 0x7f236a14d4df "cirrus" pid_file = 0x0 incoming = 0x0 show_vnc_port = 0 defconfig = <optimized out> userconfig = 6 log_mask = <optimized out> log_file = 0x0 mem_trace = {malloc = 0x7f2369ffd720 <malloc_and_trace>, realloc = 0x7f2369ffd700 <realloc_and_trace>, free = 0x7f2369ffd6f0 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} trace_events = 0x0 trace_file = 0x0 __PRETTY_FUNCTION__ = "main" args = {machine = 0x7f236a4f63c0 <pc_machine_rhel700>, ram_size = 2147483648, boot_device = 0x7f236a122806 "cad", kernel_filename = 0x0, kernel_cmdline = 0x7f236a16a8e0 "", initrd_filename = 0x0, cpu_model = 0x7fff8f939306 "host"} (gdb)
The orignial QMP command line(__com.redhat_drive_add/device_add) did not hit this issue. e.g: {"execute":"__com.redhat_drive_add","arguments": {"file":"/home/my-data-disk.raw","format":"raw","id":"drive-data-disk","aio":"native","werror":"stop","rerror":"stop"}} {"return": {}} {"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"drive-data-disk","id":"data-disk","bus":"pci.0","scsi":"off","addr":"0x9"}} {"return": {}} Best Regards, sluo
If append the 'cache' specifying ('writeback': true|false) which did not hit this issue. - e.g.1, 'writeback': *false*. { "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio':'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}, 'cache': { 'writeback': false, 'direct': true, 'no-flush': false }}} } {"return": {}} {"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"disk1","id":"disk1","bus":"pci.0","scsi":"off","addr":"0x8"}} {"return": {}} - e.g.1, 'writeback': *true*. { "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio':'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}, 'cache': { 'writeback': true, 'direct': true, 'no-flush': false }}} } {"return": {}} {"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"disk1","id":"disk1","bus":"pci.0","scsi":"off","addr":"0x8"}} {"return": {}} Best Regards, sluo
Fixed in upstream commit c6e0bd9b. We'll get it with the 7.1 rebase.
Since this bz will be fixed in qemu2.1, I guess we should update the component to qemu-kvm-rhev. Best Regards, Junyi
Reproduce this bug as follow version: Host # uname -r 3.10.0-144.el7.x86_64 # rpm -q qemu-kvm-rhev qemu-kvm-rhev-1.5.3-60.el7_0_0.6.x86_64 Guest:rhel6.6 1.Boot guest 2.remote-viewer spice://10.66.7.55:5931 3.hut-plug using blockdev_add with 'aio': 'native' specified. { "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio': 'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}}} } {"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"disk1","id":"disk1","bus":"pci.0","scsi":"off","addr":"0x8"}} Results: Segmentation fault Program received signal SIGSEGV, Segmentation fault. 0x000055555565128c in qmp_blockdev_add (options=0x555556d853b0, errp=errp@entry=0x7fffffffca48) at blockdev.c:1771 1771 bool direct = options->cache->has_direct && options->cache->direct; Missing separate debuginfos, use: debuginfo-install alsa-lib-1.0.27.2-3.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 cyrus-sasl-lib-2.1.26-17.el7.x86_64 cyrus-sasl-md5-2.1.26-17.el7.x86_64 cyrus-sasl-plain-2.1.26-17.el7.x86_64 dbus-libs-1.6.12-8.el7.x86_64 flac-libs-1.3.0-4.el7.x86_64 glib2-2.36.3-5.el7.x86_64 glibc-2.17-55.el7.x86_64 glusterfs-api-3.4.0.59rhs-1.el7.x86_64 glusterfs-libs-3.4.0.59rhs-1.el7.x86_64 gmp-5.1.1-5.el7.x86_64 gnutls-3.1.18-8.el7.x86_64 gsm-1.0.13-11.el7.x86_64 json-c-0.11-3.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.11.3-49.el7.x86_64 libICE-1.0.8-7.el7.x86_64 libSM-1.2.1-7.el7.x86_64 libX11-1.6.0-2.1.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXext-1.3.2-2.1.el7.x86_64 libXi-1.7.2-2.1.el7.x86_64 libXtst-1.2.2-2.1.el7.x86_64 libaio-0.3.109-12.el7.x86_64 libasyncns-0.8-7.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-4.el7.x86_64 libdb-5.3.21-17.el7.x86_64 libgcc-4.8.2-16.el7.x86_64 libgcrypt-1.5.3-4.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibverbs-1.1.7-6.el7.x86_64 libiscsi-1.9.0-6.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libnl-1.1.4-3.el7.x86_64 libogg-1.3.0-7.el7.x86_64 libpng-1.5.13-5.el7.x86_64 librdmacm-1.0.17.1-1.el7.x86_64 libseccomp-2.1.1-2.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libsndfile-1.0.25-9.el7.x86_64 libtasn1-3.3-3.el7.x86_64 libusbx-1.0.15-4.el7.x86_64 libuuid-2.23.2-16.el7.x86_64 libvorbis-1.3.3-8.el7.x86_64 libxcb-1.9-5.el7.x86_64 nettle-2.7.1-2.el7.x86_64 nspr-4.10.2-4.el7.x86_64 nss-3.15.4-6.el7.x86_64 nss-softokn-freebl-3.15.4-2.el7.x86_64 nss-util-3.15.4-2.el7.x86_64 openssl-libs-1.0.1e-34.el7.x86_64 p11-kit-0.18.7-4.el7.x86_64 pcre-8.32-12.el7.x86_64 pixman-0.32.4-3.el7.x86_64 pulseaudio-libs-3.0-22.el7.x86_64 tcp_wrappers-libs-7.6-77.el7.x86_64 usbredir-0.6-7.el7.x86_64 xz-libs-5.1.2-8alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64 (gdb) brt Undefined command: "brt". Try "help". (gdb) bt #0 0x000055555565128c in qmp_blockdev_add (options=0x555556d853b0, errp=errp@entry=0x7fffffffca48) at blockdev.c:1771 #1 0x000055555571b225 in qmp_marshal_input_blockdev_add ( mon=<optimized out>, qdict=<optimized out>, ret=<optimized out>) at qmp-marshal.c:3893 #2 0x00005555557a2c47 in qmp_call_cmd (cmd=<optimized out>, params=0x55555682a870, mon=0x555556515930) at /usr/src/debug/qemu-1.5.3/monitor.c:4509 #3 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4575 #4 0x000055555584f7a2 in json_message_process_token ( lexer=0x5555564f7330, token=0x555556d850d0, type=JSON_OPERATOR, x=172, y=3) at qobject/json-streamer.c:87 #5 0x000055555585ed2f in json_lexer_feed_char ( lexer=lexer@entry=0x5555564f7330, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 #6 0x000055555585edfe in json_lexer_feed (lexer=0x5555564f7330, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 #7 0x000055555584f939 in json_message_parser_feed ( parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 #8 0x00005555557a1993 in monitor_control_read (opaque=<optimized out>, ---Type <return> to continue, or q <return> to quit--- buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4596 #9 0x000055555570f541 in qemu_chr_be_write (len=<optimized out>, buf=0x7fffffffcc50 "}\210NVUU", s=0x5555564e8d70) at qemu-char.c:167 #10 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x5555564e8d70) at qemu-char.c:2492 #11 0x00007ffff74edac6 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 #12 0x00005555556e1a4a in glib_pollfds_poll () at main-loop.c:187 #13 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232 #14 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464 #15 0x0000555555602f50 in main_loop () at vl.c:1988 #16 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4359 Verify this bug as follow version: Host: # uname -r 3.10.0-144.el7.x86_64 # rpm -q qemu-kvm-rhev qemu-kvm-rhev-2.1.0-2.el7.x86_64 Guest:rhel7 Steps as same as reproduce Results: { "execute": "blockdev-add", "arguments": {'options' : {'driver': 'raw', 'id':'disk1', 'aio': 'native', 'file': {'driver': 'file', 'filename': '/home/my-data-disk.raw'}}} } {"error": {"class": "GenericError", "desc": "aio=native requires cache.direct=true"}} Forbit add block without cache paramter if with aio. According to above test ,this bug has been fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0624.html