Bug 1671266 - Qemu coredump when remove a persistent bitmap after vm re-start(dataplane enabled)
Summary: Qemu coredump when remove a persistent bitmap after vm re-start(dataplane ena...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 8.1
Assignee: John Snow
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-31 09:33 UTC by aihua liang
Modified: 2019-11-06 07:13 UTC (History)
10 users (show)

Fixed In Version: qemu-kvm-4.1.0-1.module+el8.1.0+3966+4a23dca1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-06 07:12:49 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 0 None None None 2019-11-06 07:13:21 UTC

Description aihua liang 2019-01-31 09:33:20 UTC
Description of problem:
  Qemu coredump when remove a persistent bitmap after vm re-start.

Version-Release number of selected component (if applicable):
  kernel version:4.18.0-62.el8.x86_64
  qemu-kvm version:qemu-kvm-3.1.0-10.module+el8+2732+3228f155.x86_64

How reproducible:
  100%


Steps to Reproduce:
1.Start guest with cmds:
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190123-032240-rOoB4cgD,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190123-032240-rOoB4cgD,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id8Ec4Bn  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190123-032240-rOoB4cgD,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190123-032240-rOoB4cgD,path=/var/tmp/seabios-20190123-032240-rOoB4cgD,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190123-032240-rOoB4cgD,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 \
    -blockdev driver=file,node-name=file_base,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2,auto-read-only=on \
    -blockdev driver=qcow2,file=file_base,node-name=drive_image1,auto-read-only=on \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:39:3a:3b:3c:3d,id=id1JNQsL,vectors=4,netdev=idVpZZ6A,bus=pci.0,addr=0x4  \
    -netdev tap,id=idVpZZ6A,vhost=on \
    -m 7168  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

2. Create a persistent bitmap on drive_image1
   { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true}}

3. Check bitmap info:
  {"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580104192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 7864320}], "type": "unknown"}]}}

4. Quit vm.
  (qemu)quit

5. Re-start vm, check its bitmap info:
    {"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580104192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 80084992}], "type": "unknown"}]}

6. Remove bitmap1
   { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}}

Actual results:
After step6, qemu coredump with info:
 (qemu) qemu: qemu_mutex_unlock_impl: Operation not permitted
tt.txt: line 36:  5819 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190123-032240-rOoB4cgD,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control ...


(gdb) bt
#0  0x00007fd56bc1893f in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fd56bc02c95 in __GI_abort () at abort.c:79
#2  0x00005617f388ec6e in error_exit (err=<optimized out>, msg=msg@entry=0x5617f3a17860 <__func__.19021> "qemu_mutex_unlock_impl")
    at util/qemu-thread-posix.c:36
#3  0x00005617f388eeaa in qemu_mutex_unlock_impl
    (mutex=mutex@entry=0x5617f4800b90, file=file@entry=0x5617f3a16d5f "util/async.c", line=line@entry=516) at util/qemu-thread-posix.c:96
#4  0x00005617f388a119 in aio_context_release (ctx=ctx@entry=0x5617f4800b30) at util/async.c:516
#5  0x00005617f38142c8 in bdrv_prwv_co
    (child=child@entry=0x5617f47a1800, offset=offset@entry=4286251008, qiov=qiov@entry=0x7ffd63b78a70, is_write=is_write@entry=false, flags=flags@entry=0) at block/io.c:834
#6  0x00005617f38145ea in bdrv_preadv (qiov=0x7ffd63b78a70, offset=4286251008, child=0x5617f47a1800) at block/io.c:963
#7  0x00005617f38145ea in bdrv_pread (child=0x5617f47a1800, offset=offset@entry=4286251008, buf=buf@entry=0x5617f491eec0, bytes=bytes@entry=32)
    at block/io.c:963
#8  0x00005617f37f4d47 in bitmap_list_load (bs=bs@entry=0x5617f4826620, offset=4286251008, size=32, errp=errp@entry=0x7ffd63b78b60)
    at block/qcow2-bitmap.c:565
#9  0x00005617f37f5a85 in qcow2_remove_persistent_dirty_bitmap (bs=0x5617f4826620, name=0x5617f491e5e0 "bitmap1", errp=0x7ffd63b78b60)
    at block/qcow2-bitmap.c:1284
#10 0x00005617f36759c2 in qmp_block_dirty_bitmap_remove (node=<optimized out>, name=0x5617f491e5e0 "bitmap1", errp=errp@entry=0x7ffd63b78b98)
    at blockdev.c:2903
#11 0x00005617f36823e8 in qmp_marshal_block_dirty_bitmap_remove (args=<optimized out>, ret=<optimized out>, errp=0x7ffd63b78c08)
    at qapi/qapi-commands-block-core.c:627
#12 0x00005617f387e9a3 in do_qmp_dispatch
    (errp=0x7ffd63b78c00, allow_oob=<optimized out>, request=<optimized out>, cmds=0x5617f40be990 <qmp_commands>) at qapi/qmp-dispatch.c:129
#13 0x00005617f387e9a3 in qmp_dispatch (cmds=0x5617f40be990 <qmp_commands>, request=<optimized out>, allow_oob=<optimized out>)
    at qapi/qmp-dispatch.c:171
#14 0x00005617f3588cb3 in monitor_qmp_dispatch (mon=0x5617f482ab10, req=<optimized out>, id=0x0)
    at /usr/src/debug/qemu-kvm-3.1.0-10.module+el8+2732+3228f155.x86_64/monitor.c:4085
#15 0x00005617f358ec28 in monitor_qmp_bh_dispatcher (data=<optimized out>)
    at /usr/src/debug/qemu-kvm-3.1.0-10.module+el8+2732+3228f155.x86_64/monitor.c:4157
#16 0x00005617f3889976 in aio_bh_call (bh=0x5617f477e6d0) at util/async.c:118
#17 0x00005617f3889976 in aio_bh_poll (ctx=ctx@entry=0x5617f477d380) at util/async.c:118
#18 0x00005617f388cca4 in aio_dispatch (ctx=0x5617f477d380) at util/aio-posix.c:440
#19 0x00005617f3889852 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261
#20 0x00007fd5702b689d in g_main_dispatch (context=0x5617f47f39e0) at gmain.c:3176
#21 0x00007fd5702b689d in g_main_context_dispatch (context=context@entry=0x5617f47f39e0) at gmain.c:3829
#22 0x00005617f388bf28 in glib_pollfds_poll () at util/main-loop.c:215
--Type <RET> for more, q to quit, c to continue without paging--
#23 0x00005617f388bf28 in os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:238
#24 0x00005617f388bf28 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:497
#25 0x00005617f367efc9 in main_loop () at vl.c:1910
#26 0x00005617f353f544 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4681


Expected results:
 The persistent bitmap can be removed.

Additional info:
 1. I can remove the persistent bitmap as following operations:
   start vm--> add persistent bitmap--> remove bitmap

 2. When disable dataplane, don't hit this issue.

Comment 4 Gu Nini 2019-02-26 10:52:46 UTC
Reproduced the bug on following sw versions:

Host kernel: 4.18.0-71.el8.ppc64le
qemu-kvm-3.1.0-15.module+el8+2792+e33e01a0.ppc64le

Comment 5 John Snow 2019-07-08 20:37:25 UTC
Expected to be fixed by rebase to 4.0+ with the inclusion of:
0a6c86d024c52b1e66d4f7ec01a3bb8ea2600145 blockdev: acquire aio_context for bitmap add/remove

Just like BZ #1672010

Comment 7 aihua liang 2019-08-16 03:18:42 UTC
Verified it on qemu-kvm-4.1.0-1.module+el8.1.0+3966+4a23dca1.x86_64, the problem has been resolved, so set bug's status to "Verified".

Test steps:
  1.Start guest with cmds:
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190123-032240-rOoB4cgD,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190123-032240-rOoB4cgD,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id8Ec4Bn  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190123-032240-rOoB4cgD,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190123-032240-rOoB4cgD,path=/var/tmp/seabios-20190123-032240-rOoB4cgD,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190123-032240-rOoB4cgD,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 \
    -blockdev driver=file,node-name=file_base,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2,auto-read-only=on \
    -blockdev driver=qcow2,file=file_base,node-name=drive_image1,auto-read-only=on \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:39:3a:3b:3c:3d,id=id1JNQsL,vectors=4,netdev=idVpZZ6A,bus=pci.0,addr=0x4  \
    -netdev tap,id=idVpZZ6A,vhost=on \
    -m 7168  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

2. Create a persistent bitmap on drive_image1
   { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true}}

3. Check bitmap info:
  {"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580104192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 7864320}], "type": "unknown"}]}}

4. Quit vm.
  (qemu)quit

5. Re-start vm, check its bitmap info:
    {"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580104192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 80084992}], "type": "unknown"}]}

6. Remove bitmap1
   { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}}
{"return": {}}

7. Check bitmap info:
   {'execute':'query-block'}
  no bitmap info displayed.

8. Reset vm
   {'execute':'system_reset'}
{"timestamp": {"seconds": 1565925230, "microseconds": 976963}, "event": "RESET", "data": {"guest": false, "reason": "host-qmp-system-reset"}}
{"timestamp": {"seconds": 1565925231, "microseconds": 16602}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}
{"timestamp": {"seconds": 1565925231, "microseconds": 17986}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}
  
   VM restart successfully after step8.

Comment 9 errata-xmlrpc 2019-11-06 07:12:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723


Note You need to log in before you can comment on or make changes to this bug.