Bug 1672010 - [RHEL7]Qemu coredump when remove a persistent bitmap after vm re-start(dataplane enabled)
Summary: [RHEL7]Qemu coredump when remove a persistent bitmap after vm re-start(datapl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.7
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: ---
Assignee: John Snow
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-03 04:38 UTC by aihua liang
Modified: 2019-08-22 09:20 UTC (History)
6 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-27.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-22 09:19:59 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:2553 None None None 2019-08-22 09:20:49 UTC

Description aihua liang 2019-02-03 04:38:02 UTC
Description of problem:
 Qemu coredump when remove a persistent bitmap after vm re-start(dataplane enabled)

Version-Release number of selected component (if applicable):
 kernel version:3.10.0-957.1.3.el7.x86_64
 qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-21.el7.x86_64

How reproducible:
 100%

Steps to Reproduce:
1.Start guest with qemu cmds:
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190111-062058-PzYcVzPD,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190111-062058-PzYcVzPD,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idKlOyb8  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190111-062058-PzYcVzPD,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190111-062058-PzYcVzPD,path=/var/tmp/seabios-20190111-062058-PzYcVzPD,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190111-062058-PzYcVzPD,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=scsi0,iothread=iothread0 \
    -blockdev node-name=file_node,driver=file,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2,auto-read-only=on \
    -blockdev node-name=drive_image1,file=file_node,driver=qcow2,auto-read-only=on \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,bus=scsi0.0 \
    -device virtio-net-pci,mac=9a:bd:be:bf:c0:c1,id=id0tbMEZ,vectors=4,netdev=idSMuUwi,bus=pci.0  \
    -netdev tap,id=idSMuUwi,vhost=on \
    -m 4096  \
    -smp 16,maxcpus=16,cores=8,threads=1,sockets=2  \
    -cpu 'IvyBridge',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

2.Create a persistent bitmap on drive_image1
   { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true}}

3.Check bitmap info:
   {"execute":"query-block"}
{"return": [{"device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580694016, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 61997056}], "type": "unknown"}]

4.Quit vm
  (qemu)quit

5.Re-start vm,check its bitmap info:
  {"execute":"query-block"}
{"return": [{"device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580694016, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 107282432}], "type": "unknown"}]}

6.Remove bitmap1
  { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}}
  Ncat: Connection reset by peer.

Actual results:
 After step6, qemu coredump with info:
   (qemu) qemu: qemu_mutex_unlock_impl: Operation not permitted
aliang.txt: line 33: 369329 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 ...

coredump info:
 (gdb) bt full
#0  0x00007febc60ca207 in raise () at /lib64/libc.so.6
#1  0x00007febc60cb8f8 in abort () at /lib64/libc.so.6
#2  0x000055d74853b3bf in error_exit (err=<optimized out>, msg=msg@entry=0x55d748a33ca0 <__func__.18625> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36
#3  0x000055d74889f62f in qemu_mutex_unlock_impl (mutex=mutex@entry=0x55d74a19a6a0, file=file@entry=0x55d748a3327f "util/async.c", line=line@entry=507) at util/qemu-thread-posix.c:97
        err = <optimized out>
        __PRETTY_FUNCTION__ = "qemu_mutex_unlock_impl"
        __func__ = "qemu_mutex_unlock_impl"
#4  0x000055d74889ae15 in aio_context_release (ctx=ctx@entry=0x55d74a19a640) at util/async.c:507
#5  0x000055d748819748 in bdrv_prwv_co (child=child@entry=0x55d74a08bea0, offset=offset@entry=4290445312, qiov=qiov@entry=0x7ffe580ce190, is_write=is_write@entry=false, flags=flags@entry=0) at block/io.c:830
        waited_ = <optimized out>
        ctx_ = 0x55d74a19a640
        bs_ = <optimized out>
        co = <optimized out>
        rwco = {child = 0x55d74a08bea0, offset = 4290445312, qiov = 0x7ffe580ce190, is_write = false, ret = 2147483647, flags = 0}
        __PRETTY_FUNCTION__ = "bdrv_prwv_co"
#6  0x000055d748819a56 in bdrv_pread (qiov=0x7ffe580ce190, offset=4290445312, child=0x55d74a08bea0) at block/io.c:938
        ret = <optimized out>
        qiov = {iov = 0x7ffe580ce180, niov = 1, nalloc = -1, size = 32}
        iov = {iov_base = 0x55d74b462940, iov_len = 32}
#7  0x000055d748819a56 in bdrv_pread (child=0x55d74a08bea0, offset=offset@entry=4290445312, buf=buf@entry=0x55d74b462940, bytes=bytes@entry=32) at block/io.c:959
        qiov = {iov = 0x7ffe580ce180, niov = 1, nalloc = -1, size = 32}
        iov = {iov_base = 0x55d74b462940, iov_len = 32}
#8  0x000055d7487f79e1 in bitmap_list_load (bs=0x55d74a1d1400, offset=4290445312, size=32, errp=errp@entry=0x7ffe580ce270) at block/qcow2-bitmap.c:565
        ret = <optimized out>
        s = 0x55d74a0145a0
        dir = 0x55d74b462940 ""
        dir_end = 0x55d74b462960 "\200+FK\327U"
        e = <optimized out>
        nb_dir_entries = 0
        bm_list = 0x0
        __func__ = "bitmap_list_load"
#9  0x000055d7487f86b4 in qcow2_remove_persistent_dirty_bitmap (bs=0x55d74a1d1400, name=0x55d74a5a7650 "bitmap1", errp=0x7ffe580ce270) at block/qcow2-bitmap.c:1284
        ret = <optimized out>
        s = <optimized out>
        bm = <optimized out>
        bm_list = <optimized out>
        __func__ = "qcow2_remove_persistent_dirty_bitmap"
#10 0x000055d748669662 in qmp_block_dirty_bitmap_remove (node=<optimized out>, name=0x55d74a5a7650 "bitmap1", errp=errp@entry=0x7ffe580ce2a8) at blockdev.c:3015
        bs = 0x55d74a1d1400
        bitmap = 0x55d74a08f980
        local_err = 0x0
        __func__ = "qmp_block_dirty_bitmap_remove"
#11 0x000055d7486761b0 in qmp_marshal_block_dirty_bitmap_remove (args=<optimized out>, ret=<optimized out>, errp=0x7ffe580ce318) at qapi/qapi-commands-block-core.c:627
        err = 0x0
        v = 0x55d74a00e9c0
        arg = {node = 0x55d74a056e00 "drive_image1", name = 0x55d74a5a7650 "bitmap1"}
---Type <return> to continue, or q <return> to quit---
#12 0x000055d74889082a in qmp_dispatch (errp=0x7ffe580ce310, request=0x7ffe580ce310, cmds=<optimized out>) at qapi/qmp-dispatch.c:111
        local_err = 0x0
        command = 0x55d74a1b20c0 "block-dirty-bitmap-remove"
        dict = <optimized out>
        ret = 0x0
        args = 0x55d74a1d9200
        cmd = 0x55d74a183ec0
        err = 0x0
        rsp = <optimized out>
#13 0x000055d74889082a in qmp_dispatch (cmds=<optimized out>, request=request@entry=0x55d74a1dc800) at qapi/qmp-dispatch.c:160
        err = 0x0
        rsp = <optimized out>
#14 0x000055d74858b321 in monitor_qmp_dispatch_one (req_obj=<optimized out>) at /usr/src/debug/qemu-2.12.0/monitor.c:4102
        mon = 0x55d74a06e000
        old_mon = 0x0
        req = 0x55d74a1dc800
        rsp = 0x0
        id = 0x0
        need_resume = true
#15 0x000055d74858b585 in monitor_qmp_bh_dispatcher (data=<optimized out>) at /usr/src/debug/qemu-2.12.0/monitor.c:4160
        req_obj = 0x55d74b462940
#16 0x000055d74889a771 in aio_bh_poll (bh=0x55d74a06d650) at util/async.c:90
        bh = <optimized out>
        bhp = <optimized out>
        next = 0x55d74a1828a0
        ret = 1
        deleted = false
#17 0x000055d74889a771 in aio_bh_poll (ctx=ctx@entry=0x55d74a19a3c0) at util/async.c:118
        bh = <optimized out>
        bhp = <optimized out>
        next = 0x55d74a1828a0
        ret = 1
        deleted = false
#18 0x000055d74889d820 in aio_dispatch (ctx=0x55d74a19a3c0) at util/aio-posix.c:440
#19 0x000055d74889a64e in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261
        ctx = <optimized out>
#20 0x00007febcc76c049 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#21 0x000055d74889cb17 in main_loop_wait () at util/main-loop.c:215
        context = 0x55d74a00b8c0
        pfds = <optimized out>
        context = 0x55d74a00b8c0
        ret = 1
        ret = 1
        timeout = 4294967295
        timeout_ns = 2809010079
#22 0x000055d74889cb17 in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:238
        context = 0x55d74a00b8c0
---Type <return> to continue, or q <return> to quit---
        ret = 1
        ret = 1
        timeout = 4294967295
        timeout_ns = 2809010079
#23 0x000055d74889cb17 in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:497
        ret = 1
        timeout = 4294967295
        timeout_ns = 2809010079
#24 0x000055d74853ede7 in main () at vl.c:1963
        i = <optimized out>
        snapshot = <optimized out>
        linux_boot = <optimized out>
        initrd_filename = <optimized out>
        kernel_filename = <optimized out>
        kernel_cmdline = <optimized out>
        boot_order = <optimized out>
        boot_once = 0x55d749fd49a0 "c"
        ds = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        icount_opts = <optimized out>
        accel_opts = <optimized out>
        olist = <optimized out>
        optind = 61
        optarg = 0x7ffe580cf4c0 "tcp:0:3000,server,nowait"
        loadvm = <optimized out>
        machine_class = 0x0
        cpu_model = <optimized out>
        vga_model = 0x0
        qtest_chrdev = <optimized out>
        qtest_log = <optimized out>
        pid_file = <optimized out>
        incoming = <optimized out>
        userconfig = <optimized out>
        nographic = <optimized out>
        display_remote = <optimized out>
        log_mask = <optimized out>
        log_file = <optimized out>
        trace_file = <optimized out>
        maxram_size = <optimized out>
        ram_slots = <optimized out>
        vmstate_dump_file = <optimized out>
        main_loop_err = 0x0
        err = 0x0
        list_data_dirs = <optimized out>
        dir = <optimized out>
        dirs = 0x0
---Type <return> to continue, or q <return> to quit---
        bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffe580ce5f0}
        __func__ = "main"
        __FUNCTION__ = "main"
#25 0x000055d74853ede7 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4768
        i = <optimized out>
        snapshot = <optimized out>
        linux_boot = <optimized out>
        initrd_filename = <optimized out>
        kernel_filename = <optimized out>
        kernel_cmdline = <optimized out>
        boot_order = <optimized out>
        boot_once = 0x55d749fd49a0 "c"
        ds = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        icount_opts = <optimized out>
        accel_opts = <optimized out>
        olist = <optimized out>
        optind = 61
        optarg = 0x7ffe580cf4c0 "tcp:0:3000,server,nowait"
        loadvm = <optimized out>
        machine_class = 0x0
        cpu_model = <optimized out>
        vga_model = 0x0
        qtest_chrdev = <optimized out>
        qtest_log = <optimized out>
        pid_file = <optimized out>
        incoming = <optimized out>
        userconfig = <optimized out>
        nographic = <optimized out>
        display_remote = <optimized out>
        log_mask = <optimized out>
        log_file = <optimized out>
        trace_file = <optimized out>
        maxram_size = <optimized out>
        ram_slots = <optimized out>
        vmstate_dump_file = <optimized out>
        main_loop_err = 0x0
        err = 0x0
        list_data_dirs = <optimized out>
        dir = <optimized out>
        dirs = 0x0
        bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffe580ce5f0}
        __func__ = "main"
        __FUNCTION__ = "main"



Expected results:
  The persistent bitmap can be removed.

Additional info:
  Bugs on RHEL8:
    https://bugzilla.redhat.com/show_bug.cgi?id=1671266
  And also hit this bug on qemu-kvm-rhev-2.12.0-18.el7_6.3.x86_64

Comment 2 John Snow 2019-02-06 17:02:29 UTC
Great report, thank you! Found and fixed upstream, code will be sent to list shortly with easy backports for our downstream branches.

Comment 3 John Snow 2019-02-06 17:53:31 UTC
Pending upstream: https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01614.html

Comment 5 Miroslav Rezanina 2019-04-25 03:52:07 UTC
Fix included in qemu-kvm-rhev-2.12.0-27.el7

Comment 6 aihua liang 2019-04-25 08:12:06 UTC
Verified it in qemu-kvm-rhev-2.12.0-27.el7, it has been fixed. Will set bug's status to "Verified" after it turned to "ON_QA".

Test steps:
  1.Start guest by qemu cmds:
    /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190412-011839-sZJY2UtD,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190412-011839-sZJY2UtD,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idbSDNv4  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190412-011839-sZJY2UtD,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190412-011839-sZJY2UtD,path=/var/tmp/seabios-20190412-011839-sZJY2UtD,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190412-011839-sZJY2UtD,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4,iothread=iothread0 \
    -blockdev driver=file,node-name=file_node,filename=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2 \
    -blockdev driver=qcow2,node-name=drive_image1,file=file_node \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:dc:dd:de:df:e0,id=idb7yKIW,vectors=4,netdev=idpOQ1y9,bus=pci.0,addr=0x5  \
    -netdev tap,id=idpOQ1y9,vhost=on \
    -m 30720  \
    -smp 16,maxcpus=16,cores=8,threads=1,sockets=2  \
    -cpu 'IvyBridge',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

  2.Add persistent bitmap to "drive_image1"
     { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true,"disabled":true,"granularity":4096}}
     { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap10","persistent":true}}
  
  3.Check image info
     {"execute":"query-block"}
       {"name": "bitmap1", "recording": false, "persistent": true, "busy": false, "status": "disabled", "granularity": 4096, "count": 0},{"name": "bitmap10", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 162201600}], "type": "unknown"}]}

  4.Shutdown vm, check image info
     [root@hp-z620-03 images]# qemu-img info rhel77-64-virtio-scsi.qcow2 
image: rhel77-64-virtio-scsi.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 4.3G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: bitmap10
            granularity: 65536
        [1]:
            flags:
            name: bitmap1
            granularity: 4096
    refcount bits: 16
    corrupt: false

  5.Re-start vm, then remove bitmaps
    { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}}
    { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap10'}}

  6.Check image info:
    {'execute':'query-block'}
       No bitmap info displayed.

  7.Check image info online:
     #qemu-img info rhel77-64-virtio-scsi.qcow2 -U
       No bitmap info displayed.

Comment 8 aihua liang 2019-04-28 02:27:16 UTC
As comment 6, set bug's status to "Verified".

Comment 10 errata-xmlrpc 2019-08-22 09:19:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553


Note You need to log in before you can comment on or make changes to this bug.