Description of problem: Qemu coredump when remove a persistent bitmap after vm re-start. Version-Release number of selected component (if applicable): kernel version:4.18.0-62.el8.x86_64 qemu-kvm version:qemu-kvm-3.1.0-10.module+el8+2732+3228f155.x86_64 How reproducible: 100% Steps to Reproduce: 1.Start guest with cmds: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190123-032240-rOoB4cgD,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190123-032240-rOoB4cgD,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=id8Ec4Bn \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190123-032240-rOoB4cgD,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20190123-032240-rOoB4cgD,path=/var/tmp/seabios-20190123-032240-rOoB4cgD,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20190123-032240-rOoB4cgD,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 \ -blockdev driver=file,node-name=file_base,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2,auto-read-only=on \ -blockdev driver=qcow2,file=file_base,node-name=drive_image1,auto-read-only=on \ -device scsi-hd,id=image1,drive=drive_image1 \ -device virtio-net-pci,mac=9a:39:3a:3b:3c:3d,id=id1JNQsL,vectors=4,netdev=idVpZZ6A,bus=pci.0,addr=0x4 \ -netdev tap,id=idVpZZ6A,vhost=on \ -m 7168 \ -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \ -cpu 'Skylake-Client',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ 2. Create a persistent bitmap on drive_image1 { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true}} 3. Check bitmap info: {"execute":"query-block"} {"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580104192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 7864320}], "type": "unknown"}]}} 4. Quit vm. (qemu)quit 5. Re-start vm, check its bitmap info: {"execute":"query-block"} {"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580104192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 80084992}], "type": "unknown"}]} 6. Remove bitmap1 { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}} Actual results: After step6, qemu coredump with info: (qemu) qemu: qemu_mutex_unlock_impl: Operation not permitted tt.txt: line 36: 5819 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190123-032240-rOoB4cgD,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control ... (gdb) bt #0 0x00007fd56bc1893f in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007fd56bc02c95 in __GI_abort () at abort.c:79 #2 0x00005617f388ec6e in error_exit (err=<optimized out>, msg=msg@entry=0x5617f3a17860 <__func__.19021> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36 #3 0x00005617f388eeaa in qemu_mutex_unlock_impl (mutex=mutex@entry=0x5617f4800b90, file=file@entry=0x5617f3a16d5f "util/async.c", line=line@entry=516) at util/qemu-thread-posix.c:96 #4 0x00005617f388a119 in aio_context_release (ctx=ctx@entry=0x5617f4800b30) at util/async.c:516 #5 0x00005617f38142c8 in bdrv_prwv_co (child=child@entry=0x5617f47a1800, offset=offset@entry=4286251008, qiov=qiov@entry=0x7ffd63b78a70, is_write=is_write@entry=false, flags=flags@entry=0) at block/io.c:834 #6 0x00005617f38145ea in bdrv_preadv (qiov=0x7ffd63b78a70, offset=4286251008, child=0x5617f47a1800) at block/io.c:963 #7 0x00005617f38145ea in bdrv_pread (child=0x5617f47a1800, offset=offset@entry=4286251008, buf=buf@entry=0x5617f491eec0, bytes=bytes@entry=32) at block/io.c:963 #8 0x00005617f37f4d47 in bitmap_list_load (bs=bs@entry=0x5617f4826620, offset=4286251008, size=32, errp=errp@entry=0x7ffd63b78b60) at block/qcow2-bitmap.c:565 #9 0x00005617f37f5a85 in qcow2_remove_persistent_dirty_bitmap (bs=0x5617f4826620, name=0x5617f491e5e0 "bitmap1", errp=0x7ffd63b78b60) at block/qcow2-bitmap.c:1284 #10 0x00005617f36759c2 in qmp_block_dirty_bitmap_remove (node=<optimized out>, name=0x5617f491e5e0 "bitmap1", errp=errp@entry=0x7ffd63b78b98) at blockdev.c:2903 #11 0x00005617f36823e8 in qmp_marshal_block_dirty_bitmap_remove (args=<optimized out>, ret=<optimized out>, errp=0x7ffd63b78c08) at qapi/qapi-commands-block-core.c:627 #12 0x00005617f387e9a3 in do_qmp_dispatch (errp=0x7ffd63b78c00, allow_oob=<optimized out>, request=<optimized out>, cmds=0x5617f40be990 <qmp_commands>) at qapi/qmp-dispatch.c:129 #13 0x00005617f387e9a3 in qmp_dispatch (cmds=0x5617f40be990 <qmp_commands>, request=<optimized out>, allow_oob=<optimized out>) at qapi/qmp-dispatch.c:171 #14 0x00005617f3588cb3 in monitor_qmp_dispatch (mon=0x5617f482ab10, req=<optimized out>, id=0x0) at /usr/src/debug/qemu-kvm-3.1.0-10.module+el8+2732+3228f155.x86_64/monitor.c:4085 #15 0x00005617f358ec28 in monitor_qmp_bh_dispatcher (data=<optimized out>) at /usr/src/debug/qemu-kvm-3.1.0-10.module+el8+2732+3228f155.x86_64/monitor.c:4157 #16 0x00005617f3889976 in aio_bh_call (bh=0x5617f477e6d0) at util/async.c:118 #17 0x00005617f3889976 in aio_bh_poll (ctx=ctx@entry=0x5617f477d380) at util/async.c:118 #18 0x00005617f388cca4 in aio_dispatch (ctx=0x5617f477d380) at util/aio-posix.c:440 #19 0x00005617f3889852 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261 #20 0x00007fd5702b689d in g_main_dispatch (context=0x5617f47f39e0) at gmain.c:3176 #21 0x00007fd5702b689d in g_main_context_dispatch (context=context@entry=0x5617f47f39e0) at gmain.c:3829 #22 0x00005617f388bf28 in glib_pollfds_poll () at util/main-loop.c:215 --Type <RET> for more, q to quit, c to continue without paging-- #23 0x00005617f388bf28 in os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:238 #24 0x00005617f388bf28 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:497 #25 0x00005617f367efc9 in main_loop () at vl.c:1910 #26 0x00005617f353f544 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4681 Expected results: The persistent bitmap can be removed. Additional info: 1. I can remove the persistent bitmap as following operations: start vm--> add persistent bitmap--> remove bitmap 2. When disable dataplane, don't hit this issue.
Reproduced the bug on following sw versions: Host kernel: 4.18.0-71.el8.ppc64le qemu-kvm-3.1.0-15.module+el8+2792+e33e01a0.ppc64le
Expected to be fixed by rebase to 4.0+ with the inclusion of: 0a6c86d024c52b1e66d4f7ec01a3bb8ea2600145 blockdev: acquire aio_context for bitmap add/remove Just like BZ #1672010
Verified it on qemu-kvm-4.1.0-1.module+el8.1.0+3966+4a23dca1.x86_64, the problem has been resolved, so set bug's status to "Verified". Test steps: 1.Start guest with cmds: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190123-032240-rOoB4cgD,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190123-032240-rOoB4cgD,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=id8Ec4Bn \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190123-032240-rOoB4cgD,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20190123-032240-rOoB4cgD,path=/var/tmp/seabios-20190123-032240-rOoB4cgD,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20190123-032240-rOoB4cgD,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 \ -blockdev driver=file,node-name=file_base,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2,auto-read-only=on \ -blockdev driver=qcow2,file=file_base,node-name=drive_image1,auto-read-only=on \ -device scsi-hd,id=image1,drive=drive_image1 \ -device virtio-net-pci,mac=9a:39:3a:3b:3c:3d,id=id1JNQsL,vectors=4,netdev=idVpZZ6A,bus=pci.0,addr=0x4 \ -netdev tap,id=idVpZZ6A,vhost=on \ -m 7168 \ -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \ -cpu 'Skylake-Client',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ 2. Create a persistent bitmap on drive_image1 { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true}} 3. Check bitmap info: {"execute":"query-block"} {"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580104192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 7864320}], "type": "unknown"}]}} 4. Quit vm. (qemu)quit 5. Re-start vm, check its bitmap info: {"execute":"query-block"} {"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580104192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 80084992}], "type": "unknown"}]} 6. Remove bitmap1 { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}} {"return": {}} 7. Check bitmap info: {'execute':'query-block'} no bitmap info displayed. 8. Reset vm {'execute':'system_reset'} {"timestamp": {"seconds": 1565925230, "microseconds": 976963}, "event": "RESET", "data": {"guest": false, "reason": "host-qmp-system-reset"}} {"timestamp": {"seconds": 1565925231, "microseconds": 16602}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} {"timestamp": {"seconds": 1565925231, "microseconds": 17986}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} VM restart successfully after step8.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3723