Bug 1672010
| Summary: | [RHEL7]Qemu coredump when remove a persistent bitmap after vm re-start(dataplane enabled) | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | aihua liang <aliang> |
| Component: | qemu-kvm-rhev | Assignee: | John Snow <jsnow> |
| Status: | CLOSED ERRATA | QA Contact: | aihua liang <aliang> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.7 | CC: | chaoyang, coli, juzhang, ngu, qzhang, virt-maint |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-rhev-2.12.0-27.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-08-22 09:19:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Great report, thank you! Found and fixed upstream, code will be sent to list shortly with easy backports for our downstream branches. Pending upstream: https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01614.html Fix included in qemu-kvm-rhev-2.12.0-27.el7 Verified it in qemu-kvm-rhev-2.12.0-27.el7, it has been fixed. Will set bug's status to "Verified" after it turned to "ON_QA".
Test steps:
1.Start guest by qemu cmds:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-machine pc \
-nodefaults \
-device VGA,bus=pci.0,addr=0x2 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190412-011839-sZJY2UtD,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190412-011839-sZJY2UtD,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idbSDNv4 \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190412-011839-sZJY2UtD,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20190412-011839-sZJY2UtD,path=/var/tmp/seabios-20190412-011839-sZJY2UtD,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20190412-011839-sZJY2UtD,iobase=0x402 \
-device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
-object iothread,id=iothread0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4,iothread=iothread0 \
-blockdev driver=file,node-name=file_node,filename=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2 \
-blockdev driver=qcow2,node-name=drive_image1,file=file_node \
-device scsi-hd,id=image1,drive=drive_image1 \
-device virtio-net-pci,mac=9a:dc:dd:de:df:e0,id=idb7yKIW,vectors=4,netdev=idpOQ1y9,bus=pci.0,addr=0x5 \
-netdev tap,id=idpOQ1y9,vhost=on \
-m 30720 \
-smp 16,maxcpus=16,cores=8,threads=1,sockets=2 \
-cpu 'IvyBridge',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :0 \
-rtc base=utc,clock=host,driftfix=slew \
-boot menu=off,strict=off,order=cdn,once=c \
-enable-kvm \
-monitor stdio \
-qmp tcp:0:3000,server,nowait \
2.Add persistent bitmap to "drive_image1"
{ "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true,"disabled":true,"granularity":4096}}
{ "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap10","persistent":true}}
3.Check image info
{"execute":"query-block"}
{"name": "bitmap1", "recording": false, "persistent": true, "busy": false, "status": "disabled", "granularity": 4096, "count": 0},{"name": "bitmap10", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 162201600}], "type": "unknown"}]}
4.Shutdown vm, check image info
[root@hp-z620-03 images]# qemu-img info rhel77-64-virtio-scsi.qcow2
image: rhel77-64-virtio-scsi.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 4.3G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
bitmaps:
[0]:
flags:
[0]: auto
name: bitmap10
granularity: 65536
[1]:
flags:
name: bitmap1
granularity: 4096
refcount bits: 16
corrupt: false
5.Re-start vm, then remove bitmaps
{ 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}}
{ 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap10'}}
6.Check image info:
{'execute':'query-block'}
No bitmap info displayed.
7.Check image info online:
#qemu-img info rhel77-64-virtio-scsi.qcow2 -U
No bitmap info displayed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:2553 |
Description of problem: Qemu coredump when remove a persistent bitmap after vm re-start(dataplane enabled) Version-Release number of selected component (if applicable): kernel version:3.10.0-957.1.3.el7.x86_64 qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-21.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Start guest with qemu cmds: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190111-062058-PzYcVzPD,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190111-062058-PzYcVzPD,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idKlOyb8 \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190111-062058-PzYcVzPD,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20190111-062058-PzYcVzPD,path=/var/tmp/seabios-20190111-062058-PzYcVzPD,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20190111-062058-PzYcVzPD,iobase=0x402 \ -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=scsi0,iothread=iothread0 \ -blockdev node-name=file_node,driver=file,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2,auto-read-only=on \ -blockdev node-name=drive_image1,file=file_node,driver=qcow2,auto-read-only=on \ -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,bus=scsi0.0 \ -device virtio-net-pci,mac=9a:bd:be:bf:c0:c1,id=id0tbMEZ,vectors=4,netdev=idSMuUwi,bus=pci.0 \ -netdev tap,id=idSMuUwi,vhost=on \ -m 4096 \ -smp 16,maxcpus=16,cores=8,threads=1,sockets=2 \ -cpu 'IvyBridge',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,strict=off,order=cdn,once=c \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ 2.Create a persistent bitmap on drive_image1 { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap1","persistent":true}} 3.Check bitmap info: {"execute":"query-block"} {"return": [{"device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580694016, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 61997056}], "type": "unknown"}] 4.Quit vm (qemu)quit 5.Re-start vm,check its bitmap info: {"execute":"query-block"} {"return": [{"device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 8580694016, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "image1", "dirty-bitmaps": [{"name": "bitmap1", "status": "active", "granularity": 65536, "count": 107282432}], "type": "unknown"}]} 6.Remove bitmap1 { 'execute': 'block-dirty-bitmap-remove', 'arguments': {'node':'drive_image1', 'name':'bitmap1'}} Ncat: Connection reset by peer. Actual results: After step6, qemu coredump with info: (qemu) qemu: qemu_mutex_unlock_impl: Operation not permitted aliang.txt: line 33: 369329 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 ... coredump info: (gdb) bt full #0 0x00007febc60ca207 in raise () at /lib64/libc.so.6 #1 0x00007febc60cb8f8 in abort () at /lib64/libc.so.6 #2 0x000055d74853b3bf in error_exit (err=<optimized out>, msg=msg@entry=0x55d748a33ca0 <__func__.18625> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36 #3 0x000055d74889f62f in qemu_mutex_unlock_impl (mutex=mutex@entry=0x55d74a19a6a0, file=file@entry=0x55d748a3327f "util/async.c", line=line@entry=507) at util/qemu-thread-posix.c:97 err = <optimized out> __PRETTY_FUNCTION__ = "qemu_mutex_unlock_impl" __func__ = "qemu_mutex_unlock_impl" #4 0x000055d74889ae15 in aio_context_release (ctx=ctx@entry=0x55d74a19a640) at util/async.c:507 #5 0x000055d748819748 in bdrv_prwv_co (child=child@entry=0x55d74a08bea0, offset=offset@entry=4290445312, qiov=qiov@entry=0x7ffe580ce190, is_write=is_write@entry=false, flags=flags@entry=0) at block/io.c:830 waited_ = <optimized out> ctx_ = 0x55d74a19a640 bs_ = <optimized out> co = <optimized out> rwco = {child = 0x55d74a08bea0, offset = 4290445312, qiov = 0x7ffe580ce190, is_write = false, ret = 2147483647, flags = 0} __PRETTY_FUNCTION__ = "bdrv_prwv_co" #6 0x000055d748819a56 in bdrv_pread (qiov=0x7ffe580ce190, offset=4290445312, child=0x55d74a08bea0) at block/io.c:938 ret = <optimized out> qiov = {iov = 0x7ffe580ce180, niov = 1, nalloc = -1, size = 32} iov = {iov_base = 0x55d74b462940, iov_len = 32} #7 0x000055d748819a56 in bdrv_pread (child=0x55d74a08bea0, offset=offset@entry=4290445312, buf=buf@entry=0x55d74b462940, bytes=bytes@entry=32) at block/io.c:959 qiov = {iov = 0x7ffe580ce180, niov = 1, nalloc = -1, size = 32} iov = {iov_base = 0x55d74b462940, iov_len = 32} #8 0x000055d7487f79e1 in bitmap_list_load (bs=0x55d74a1d1400, offset=4290445312, size=32, errp=errp@entry=0x7ffe580ce270) at block/qcow2-bitmap.c:565 ret = <optimized out> s = 0x55d74a0145a0 dir = 0x55d74b462940 "" dir_end = 0x55d74b462960 "\200+FK\327U" e = <optimized out> nb_dir_entries = 0 bm_list = 0x0 __func__ = "bitmap_list_load" #9 0x000055d7487f86b4 in qcow2_remove_persistent_dirty_bitmap (bs=0x55d74a1d1400, name=0x55d74a5a7650 "bitmap1", errp=0x7ffe580ce270) at block/qcow2-bitmap.c:1284 ret = <optimized out> s = <optimized out> bm = <optimized out> bm_list = <optimized out> __func__ = "qcow2_remove_persistent_dirty_bitmap" #10 0x000055d748669662 in qmp_block_dirty_bitmap_remove (node=<optimized out>, name=0x55d74a5a7650 "bitmap1", errp=errp@entry=0x7ffe580ce2a8) at blockdev.c:3015 bs = 0x55d74a1d1400 bitmap = 0x55d74a08f980 local_err = 0x0 __func__ = "qmp_block_dirty_bitmap_remove" #11 0x000055d7486761b0 in qmp_marshal_block_dirty_bitmap_remove (args=<optimized out>, ret=<optimized out>, errp=0x7ffe580ce318) at qapi/qapi-commands-block-core.c:627 err = 0x0 v = 0x55d74a00e9c0 arg = {node = 0x55d74a056e00 "drive_image1", name = 0x55d74a5a7650 "bitmap1"} ---Type <return> to continue, or q <return> to quit--- #12 0x000055d74889082a in qmp_dispatch (errp=0x7ffe580ce310, request=0x7ffe580ce310, cmds=<optimized out>) at qapi/qmp-dispatch.c:111 local_err = 0x0 command = 0x55d74a1b20c0 "block-dirty-bitmap-remove" dict = <optimized out> ret = 0x0 args = 0x55d74a1d9200 cmd = 0x55d74a183ec0 err = 0x0 rsp = <optimized out> #13 0x000055d74889082a in qmp_dispatch (cmds=<optimized out>, request=request@entry=0x55d74a1dc800) at qapi/qmp-dispatch.c:160 err = 0x0 rsp = <optimized out> #14 0x000055d74858b321 in monitor_qmp_dispatch_one (req_obj=<optimized out>) at /usr/src/debug/qemu-2.12.0/monitor.c:4102 mon = 0x55d74a06e000 old_mon = 0x0 req = 0x55d74a1dc800 rsp = 0x0 id = 0x0 need_resume = true #15 0x000055d74858b585 in monitor_qmp_bh_dispatcher (data=<optimized out>) at /usr/src/debug/qemu-2.12.0/monitor.c:4160 req_obj = 0x55d74b462940 #16 0x000055d74889a771 in aio_bh_poll (bh=0x55d74a06d650) at util/async.c:90 bh = <optimized out> bhp = <optimized out> next = 0x55d74a1828a0 ret = 1 deleted = false #17 0x000055d74889a771 in aio_bh_poll (ctx=ctx@entry=0x55d74a19a3c0) at util/async.c:118 bh = <optimized out> bhp = <optimized out> next = 0x55d74a1828a0 ret = 1 deleted = false #18 0x000055d74889d820 in aio_dispatch (ctx=0x55d74a19a3c0) at util/aio-posix.c:440 #19 0x000055d74889a64e in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261 ctx = <optimized out> #20 0x00007febcc76c049 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0 #21 0x000055d74889cb17 in main_loop_wait () at util/main-loop.c:215 context = 0x55d74a00b8c0 pfds = <optimized out> context = 0x55d74a00b8c0 ret = 1 ret = 1 timeout = 4294967295 timeout_ns = 2809010079 #22 0x000055d74889cb17 in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:238 context = 0x55d74a00b8c0 ---Type <return> to continue, or q <return> to quit--- ret = 1 ret = 1 timeout = 4294967295 timeout_ns = 2809010079 #23 0x000055d74889cb17 in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:497 ret = 1 timeout = 4294967295 timeout_ns = 2809010079 #24 0x000055d74853ede7 in main () at vl.c:1963 i = <optimized out> snapshot = <optimized out> linux_boot = <optimized out> initrd_filename = <optimized out> kernel_filename = <optimized out> kernel_cmdline = <optimized out> boot_order = <optimized out> boot_once = 0x55d749fd49a0 "c" ds = <optimized out> opts = <optimized out> machine_opts = <optimized out> icount_opts = <optimized out> accel_opts = <optimized out> olist = <optimized out> optind = 61 optarg = 0x7ffe580cf4c0 "tcp:0:3000,server,nowait" loadvm = <optimized out> machine_class = 0x0 cpu_model = <optimized out> vga_model = 0x0 qtest_chrdev = <optimized out> qtest_log = <optimized out> pid_file = <optimized out> incoming = <optimized out> userconfig = <optimized out> nographic = <optimized out> display_remote = <optimized out> log_mask = <optimized out> log_file = <optimized out> trace_file = <optimized out> maxram_size = <optimized out> ram_slots = <optimized out> vmstate_dump_file = <optimized out> main_loop_err = 0x0 err = 0x0 list_data_dirs = <optimized out> dir = <optimized out> dirs = 0x0 ---Type <return> to continue, or q <return> to quit--- bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffe580ce5f0} __func__ = "main" __FUNCTION__ = "main" #25 0x000055d74853ede7 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4768 i = <optimized out> snapshot = <optimized out> linux_boot = <optimized out> initrd_filename = <optimized out> kernel_filename = <optimized out> kernel_cmdline = <optimized out> boot_order = <optimized out> boot_once = 0x55d749fd49a0 "c" ds = <optimized out> opts = <optimized out> machine_opts = <optimized out> icount_opts = <optimized out> accel_opts = <optimized out> olist = <optimized out> optind = 61 optarg = 0x7ffe580cf4c0 "tcp:0:3000,server,nowait" loadvm = <optimized out> machine_class = 0x0 cpu_model = <optimized out> vga_model = 0x0 qtest_chrdev = <optimized out> qtest_log = <optimized out> pid_file = <optimized out> incoming = <optimized out> userconfig = <optimized out> nographic = <optimized out> display_remote = <optimized out> log_mask = <optimized out> log_file = <optimized out> trace_file = <optimized out> maxram_size = <optimized out> ram_slots = <optimized out> vmstate_dump_file = <optimized out> main_loop_err = 0x0 err = 0x0 list_data_dirs = <optimized out> dir = <optimized out> dirs = 0x0 bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffe580ce5f0} __func__ = "main" __FUNCTION__ = "main" Expected results: The persistent bitmap can be removed. Additional info: Bugs on RHEL8: https://bugzilla.redhat.com/show_bug.cgi?id=1671266 And also hit this bug on qemu-kvm-rhev-2.12.0-18.el7_6.3.x86_64