RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1672010 - [RHEL7]Qemu coredump when remove a persistent bitmap after vm re-start(dataplane enabled)
Summary: [RHEL7]Qemu coredump when remove a persistent bitmap after vm re-start(datapl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.7
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: ---
Assignee: John Snow
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-03 04:38 UTC by aihua liang
Modified: 2019-08-22 09:20 UTC (History)
6 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-27.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-22 09:19:59 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:2553 0 None None None 2019-08-22 09:20:49 UTC

Description aihua liang 2019-02-03 04:38:02 UTC
Description of problem:
 Qemu coredump when remove a persistent bitmap after vm re-start(dataplane enabled)

Version-Release number of selected component (if applicable):
 kernel version:3.10.0-957.1.3.el7.x86_64
 qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-21.el7.x86_64

How reproducible:
 100%

Steps to Reproduce:
1.Start guest with qemu cmds:
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190111-062058-PzYcVzPD,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190111-062058-PzYcVzPD,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idKlOyb8  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190111-062058-PzYcVzPD,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190111-062058-PzYcVzPD,path=/var/tmp/seabios-20190111-062058-PzYcVzPD,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190111-062058-PzYcVzPD,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=scsi0,iothread=iothread0 \
    -blockdev node-name=file_node,driver=file,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2,auto-read-only=on \
    -blockdev node-name=drive_image1,file=file_node,driver=qcow2,auto-read-only=on \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,bus=scsi0.0 \
    -device virtio-net-pci,mac=9a:bd:be:bf:c0:c1,id=id0tbMEZ,vectors=4,netdev=idSMuUwi,bus=pci.0  \
    -netdev tap,id=idSMuUwi,vhost=on \
    -m 4096  \
    -smp 16,maxcpus=16,cores=8,threads=1,sockets=2  \
    -cpu 'IvyBridge',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

2.Create a persistent bitmap on drive_image1
   { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true}}

3.Check bitmap info:
   {"execute":"query-block"}
{"return": [{"device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580694016, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 61997056}], "type": "unknown"}]

4.Quit vm
  (qemu)quit

5.Re-start vm,check its bitmap info:
  {"execute":"query-block"}
{"return": [{"device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580694016, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 107282432}], "type": "unknown"}]}

6.Remove bitmap1
  { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}}
  Ncat: Connection reset by peer.

Actual results:
 After step6, qemu coredump with info:
   (qemu) qemu: qemu_mutex_unlock_impl: Operation not permitted
aliang.txt: line 33: 369329 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 ...

coredump info:
 (gdb) bt full
#0  0x00007febc60ca207 in raise () at /lib64/libc.so.6
#1  0x00007febc60cb8f8 in abort () at /lib64/libc.so.6
#2  0x000055d74853b3bf in error_exit (err=<optimized out>, msg=msg@entry=0x55d748a33ca0 <__func__.18625> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36
#3  0x000055d74889f62f in qemu_mutex_unlock_impl (mutex=mutex@entry=0x55d74a19a6a0, file=file@entry=0x55d748a3327f "util/async.c", line=line@entry=507) at util/qemu-thread-posix.c:97
        err = <optimized out>
        __PRETTY_FUNCTION__ = "qemu_mutex_unlock_impl"
        __func__ = "qemu_mutex_unlock_impl"
#4  0x000055d74889ae15 in aio_context_release (ctx=ctx@entry=0x55d74a19a640) at util/async.c:507
#5  0x000055d748819748 in bdrv_prwv_co (child=child@entry=0x55d74a08bea0, offset=offset@entry=4290445312, qiov=qiov@entry=0x7ffe580ce190, is_write=is_write@entry=false, flags=flags@entry=0) at block/io.c:830
        waited_ = <optimized out>
        ctx_ = 0x55d74a19a640
        bs_ = <optimized out>
        co = <optimized out>
        rwco = {child = 0x55d74a08bea0, offset = 4290445312, qiov = 0x7ffe580ce190, is_write = false, ret = 2147483647, flags = 0}
        __PRETTY_FUNCTION__ = "bdrv_prwv_co"
#6  0x000055d748819a56 in bdrv_pread (qiov=0x7ffe580ce190, offset=4290445312, child=0x55d74a08bea0) at block/io.c:938
        ret = <optimized out>
        qiov = {iov = 0x7ffe580ce180, niov = 1, nalloc = -1, size = 32}
        iov = {iov_base = 0x55d74b462940, iov_len = 32}
#7  0x000055d748819a56 in bdrv_pread (child=0x55d74a08bea0, offset=offset@entry=4290445312, buf=buf@entry=0x55d74b462940, bytes=bytes@entry=32) at block/io.c:959
        qiov = {iov = 0x7ffe580ce180, niov = 1, nalloc = -1, size = 32}
        iov = {iov_base = 0x55d74b462940, iov_len = 32}
#8  0x000055d7487f79e1 in bitmap_list_load (bs=0x55d74a1d1400, offset=4290445312, size=32, errp=errp@entry=0x7ffe580ce270) at block/qcow2-bitmap.c:565
        ret = <optimized out>
        s = 0x55d74a0145a0
        dir = 0x55d74b462940 ""
        dir_end = 0x55d74b462960 "\200+FK\327U"
        e = <optimized out>
        nb_dir_entries = 0
        bm_list = 0x0
        __func__ = "bitmap_list_load"
#9  0x000055d7487f86b4 in qcow2_remove_persistent_dirty_bitmap (bs=0x55d74a1d1400, name=0x55d74a5a7650 "bitmap1", errp=0x7ffe580ce270) at block/qcow2-bitmap.c:1284
        ret = <optimized out>
        s = <optimized out>
        bm = <optimized out>
        bm_list = <optimized out>
        __func__ = "qcow2_remove_persistent_dirty_bitmap"
#10 0x000055d748669662 in qmp_block_dirty_bitmap_remove (node=<optimized out>, name=0x55d74a5a7650 "bitmap1", errp=errp@entry=0x7ffe580ce2a8) at blockdev.c:3015
        bs = 0x55d74a1d1400
        bitmap = 0x55d74a08f980
        local_err = 0x0
        __func__ = "qmp_block_dirty_bitmap_remove"
#11 0x000055d7486761b0 in qmp_marshal_block_dirty_bitmap_remove (args=<optimized out>, ret=<optimized out>, errp=0x7ffe580ce318) at qapi/qapi-commands-block-core.c:627
        err = 0x0
        v = 0x55d74a00e9c0
        arg = {node = 0x55d74a056e00 "drive_image1", name = 0x55d74a5a7650 "bitmap1"}
---Type <return> to continue, or q <return> to quit---
#12 0x000055d74889082a in qmp_dispatch (errp=0x7ffe580ce310, request=0x7ffe580ce310, cmds=<optimized out>) at qapi/qmp-dispatch.c:111
        local_err = 0x0
        command = 0x55d74a1b20c0 "block-dirty-bitmap-remove"
        dict = <optimized out>
        ret = 0x0
        args = 0x55d74a1d9200
        cmd = 0x55d74a183ec0
        err = 0x0
        rsp = <optimized out>
#13 0x000055d74889082a in qmp_dispatch (cmds=<optimized out>, request=request@entry=0x55d74a1dc800) at qapi/qmp-dispatch.c:160
        err = 0x0
        rsp = <optimized out>
#14 0x000055d74858b321 in monitor_qmp_dispatch_one (req_obj=<optimized out>) at /usr/src/debug/qemu-2.12.0/monitor.c:4102
        mon = 0x55d74a06e000
        old_mon = 0x0
        req = 0x55d74a1dc800
        rsp = 0x0
        id = 0x0
        need_resume = true
#15 0x000055d74858b585 in monitor_qmp_bh_dispatcher (data=<optimized out>) at /usr/src/debug/qemu-2.12.0/monitor.c:4160
        req_obj = 0x55d74b462940
#16 0x000055d74889a771 in aio_bh_poll (bh=0x55d74a06d650) at util/async.c:90
        bh = <optimized out>
        bhp = <optimized out>
        next = 0x55d74a1828a0
        ret = 1
        deleted = false
#17 0x000055d74889a771 in aio_bh_poll (ctx=ctx@entry=0x55d74a19a3c0) at util/async.c:118
        bh = <optimized out>
        bhp = <optimized out>
        next = 0x55d74a1828a0
        ret = 1
        deleted = false
#18 0x000055d74889d820 in aio_dispatch (ctx=0x55d74a19a3c0) at util/aio-posix.c:440
#19 0x000055d74889a64e in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261
        ctx = <optimized out>
#20 0x00007febcc76c049 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#21 0x000055d74889cb17 in main_loop_wait () at util/main-loop.c:215
        context = 0x55d74a00b8c0
        pfds = <optimized out>
        context = 0x55d74a00b8c0
        ret = 1
        ret = 1
        timeout = 4294967295
        timeout_ns = 2809010079
#22 0x000055d74889cb17 in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:238
        context = 0x55d74a00b8c0
---Type <return> to continue, or q <return> to quit---
        ret = 1
        ret = 1
        timeout = 4294967295
        timeout_ns = 2809010079
#23 0x000055d74889cb17 in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:497
        ret = 1
        timeout = 4294967295
        timeout_ns = 2809010079
#24 0x000055d74853ede7 in main () at vl.c:1963
        i = <optimized out>
        snapshot = <optimized out>
        linux_boot = <optimized out>
        initrd_filename = <optimized out>
        kernel_filename = <optimized out>
        kernel_cmdline = <optimized out>
        boot_order = <optimized out>
        boot_once = 0x55d749fd49a0 "c"
        ds = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        icount_opts = <optimized out>
        accel_opts = <optimized out>
        olist = <optimized out>
        optind = 61
        optarg = 0x7ffe580cf4c0 "tcp:0:3000,server,nowait"
        loadvm = <optimized out>
        machine_class = 0x0
        cpu_model = <optimized out>
        vga_model = 0x0
        qtest_chrdev = <optimized out>
        qtest_log = <optimized out>
        pid_file = <optimized out>
        incoming = <optimized out>
        userconfig = <optimized out>
        nographic = <optimized out>
        display_remote = <optimized out>
        log_mask = <optimized out>
        log_file = <optimized out>
        trace_file = <optimized out>
        maxram_size = <optimized out>
        ram_slots = <optimized out>
        vmstate_dump_file = <optimized out>
        main_loop_err = 0x0
        err = 0x0
        list_data_dirs = <optimized out>
        dir = <optimized out>
        dirs = 0x0
---Type <return> to continue, or q <return> to quit---
        bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffe580ce5f0}
        __func__ = "main"
        __FUNCTION__ = "main"
#25 0x000055d74853ede7 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4768
        i = <optimized out>
        snapshot = <optimized out>
        linux_boot = <optimized out>
        initrd_filename = <optimized out>
        kernel_filename = <optimized out>
        kernel_cmdline = <optimized out>
        boot_order = <optimized out>
        boot_once = 0x55d749fd49a0 "c"
        ds = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        icount_opts = <optimized out>
        accel_opts = <optimized out>
        olist = <optimized out>
        optind = 61
        optarg = 0x7ffe580cf4c0 "tcp:0:3000,server,nowait"
        loadvm = <optimized out>
        machine_class = 0x0
        cpu_model = <optimized out>
        vga_model = 0x0
        qtest_chrdev = <optimized out>
        qtest_log = <optimized out>
        pid_file = <optimized out>
        incoming = <optimized out>
        userconfig = <optimized out>
        nographic = <optimized out>
        display_remote = <optimized out>
        log_mask = <optimized out>
        log_file = <optimized out>
        trace_file = <optimized out>
        maxram_size = <optimized out>
        ram_slots = <optimized out>
        vmstate_dump_file = <optimized out>
        main_loop_err = 0x0
        err = 0x0
        list_data_dirs = <optimized out>
        dir = <optimized out>
        dirs = 0x0
        bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffe580ce5f0}
        __func__ = "main"
        __FUNCTION__ = "main"



Expected results:
  The persistent bitmap can be removed.

Additional info:
  Bugs on RHEL8:
    https://bugzilla.redhat.com/show_bug.cgi?id=1671266
  And also hit this bug on qemu-kvm-rhev-2.12.0-18.el7_6.3.x86_64

Comment 2 John Snow 2019-02-06 17:02:29 UTC
Great report, thank you! Found and fixed upstream, code will be sent to list shortly with easy backports for our downstream branches.

Comment 3 John Snow 2019-02-06 17:53:31 UTC
Pending upstream: https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01614.html

Comment 5 Miroslav Rezanina 2019-04-25 03:52:07 UTC
Fix included in qemu-kvm-rhev-2.12.0-27.el7

Comment 6 aihua liang 2019-04-25 08:12:06 UTC
Verified it in qemu-kvm-rhev-2.12.0-27.el7, it has been fixed. Will set bug's status to "Verified" after it turned to "ON_QA".

Test steps:
  1.Start guest by qemu cmds:
    /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190412-011839-sZJY2UtD,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190412-011839-sZJY2UtD,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idbSDNv4  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190412-011839-sZJY2UtD,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190412-011839-sZJY2UtD,path=/var/tmp/seabios-20190412-011839-sZJY2UtD,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190412-011839-sZJY2UtD,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4,iothread=iothread0 \
    -blockdev driver=file,node-name=file_node,filename=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2 \
    -blockdev driver=qcow2,node-name=drive_image1,file=file_node \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:dc:dd:de:df:e0,id=idb7yKIW,vectors=4,netdev=idpOQ1y9,bus=pci.0,addr=0x5  \
    -netdev tap,id=idpOQ1y9,vhost=on \
    -m 30720  \
    -smp 16,maxcpus=16,cores=8,threads=1,sockets=2  \
    -cpu 'IvyBridge',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

  2.Add persistent bitmap to "drive_image1"
     { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true,"disabled":true,"granularity":4096}}
     { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap10","persistent":true}}
  
  3.Check image info
     {"execute":"query-block"}
       {"name": "bitmap1", "recording": false, "persistent": true, "busy": false, "status": "disabled", "granularity": 4096, "count": 0},{"name": "bitmap10", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 162201600}], "type": "unknown"}]}

  4.Shutdown vm, check image info
     [root@hp-z620-03 images]# qemu-img info rhel77-64-virtio-scsi.qcow2 
image: rhel77-64-virtio-scsi.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 4.3G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: bitmap10
            granularity: 65536
        [1]:
            flags:
            name: bitmap1
            granularity: 4096
    refcount bits: 16
    corrupt: false

  5.Re-start vm, then remove bitmaps
    { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}}
    { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap10'}}

  6.Check image info:
    {'execute':'query-block'}
       No bitmap info displayed.

  7.Check image info online:
     #qemu-img info rhel77-64-virtio-scsi.qcow2 -U
       No bitmap info displayed.

Comment 8 aihua liang 2019-04-28 02:27:16 UTC
As comment 6, set bug's status to "Verified".

Comment 10 errata-xmlrpc 2019-08-22 09:19:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553


Note You need to log in before you can comment on or make changes to this bug.