Bug 1778646
| Summary: | Qemu core dump when add a persistent bitmap(data plane enable) | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | aihua liang <aliang> |
| Component: | qemu-kvm | Assignee: | John Snow <jsnow> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | aihua liang <aliang> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 8.2 | CC: | coli, jinzhao, juzhang, ngu, qzhang, virt-maint, xutian |
| Target Milestone: | rc | Keywords: | Regression, TestBlocker |
| Target Release: | 8.2 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-12-06 11:14:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Additional info: When disable data plane, not hit this issue. Core dump info located at: 10.73.194.27:/vol/s2coredump/bz1778646/core.qemu-kvm.0.e827b7d3218c403da1d4c4e721c3df06.29879.1575277198000000.lz4 Test on qemu-kvm-4.1.0-16.module+el8.1.1+4917+752cfd65.x86_64, don't hit this issue, set it's a regression. As it block all persistent bitmap test, set it as a Testblocker. (In reply to aihua liang from comment #0) > Description of problem: > Qemu core dump when add a persistent bitmap > > Version-Release number of selected component (if applicable): > kernel version:4.18.0-147.el8.x86_64 > qemu-kvm version:qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64 > > How reproducible: > 100% > > Steps to Reproduce: > 1.Start guest with qemu cmds: > /usr/libexec/qemu-kvm \ > -name 'avocado-vt-vm1' \ > -machine q35 \ > -nodefaults \ > -device VGA,bus=pcie.0,addr=0x1 \ > -m 7168 \ > -smp 4,maxcpus=4,cores=2,threads=1,dies=1,sockets=2 \ > -cpu 'Skylake-Client',+kvm_pv_unhalt \ > -chardev > socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20191118- > 011823-gEG3j1ms,server,nowait \ > -mon chardev=qmp_id_qmpmonitor1,mode=control \ > -chardev > socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20191118- > 011823-gEG3j1mt,server,nowait \ > -mon chardev=qmp_id_catch_monitor,mode=control \ > -device pvpanic,ioport=0x505,id=id4p8G4l \ > -chardev > socket,server,id=chardev_serial0,path=/var/tmp/serial-serial0-20191118- > 011823-gEG3j1mt,nowait \ > -device isa-serial,id=serial0,chardev=chardev_serial0 \ > -chardev > socket,id=seabioslog_id_20191118-011823-gEG3j1mt,path=/var/tmp/seabios- > 20191118-011823-gEG3j1mt,server,nowait \ > -device > isa-debugcon,chardev=seabioslog_id_20191118-011823-gEG3j1mt,iobase=0x402 \ > -device > pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \ > -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \ > -object iothread,id=iothread0 \ > -drive > id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2, > file=/home/kvm_autotest_root/images/rhel820-64-virtio.qcow2 \ > -device > pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \ > -device > virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pcie.0-root-port- > 3,addr=0x0,iothread=iothread0 \ > -drive > id=drive_data1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2, > file=/home/aliang/data.qcow2 \ > -device > pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \ > -device > virtio-blk-pci,id=data1,drive=drive_data1,bus=pcie.0-root-port-6,addr=0x0, > iothread=iothread0 \ > -device > pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \ > -device > virtio-net-pci,mac=9a:4f:f4:e5:bd:67,id=idkQvhgf,netdev=idnMcj5J,bus=pcie.0- > root-port-4,addr=0x0 \ > -netdev tap,id=idnMcj5J,vhost=on \ > -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ > -vnc :0 \ > -rtc base=utc,clock=host,driftfix=slew \ > -boot order=cdn,once=c,menu=off,strict=off \ > -enable-kvm \ > -device > pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie. > 0 \ > -monitor stdio \ > -qmp tcp:0:3000,server,nowait \ > -device virtio-serial-pci,id=virtio-serial0,bus=pcie.0,addr=0x8 \ > -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \ > -device > virtserialport,bus=virtio-serial0.0,chardev=qga0,id=qemu-ga0,name=org.qemu. > guest_agent.0 \ > It seems you're testing incremental backup (bitmaps) with -drive instead of -blockdev. Starting in RHEL-8.2 libvirt will default to -blockdev and we do not have expectations that incremental backup works with -drive. Can you please re-test? In the meanwhile, reassigning to John Snow. Test with -blockdev, it works ok.
1.Start guest with qemu cmds:
...
-blockdev driver=file,node-name=data_node,filename=/home/data.qcow2 \
-blockdev driver=qcow2,file=data_node,node-name=drive_data1 \
-device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
-device virtio-blk-pci,id=data1,drive=drive_data1,bus=pcie.0-root-port-6,addr=0x0 \
2. Add persistent bitmap to data disk
{ "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_data1", "name": "bitmap0","persistent":true}}
{"return": {}}
3. Query block info
{"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel820-64-virtio.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5894922240, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel820-64-virtio.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/image1/virtio-backend", "type": "unknown"}, {"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 2147483648, "filename": "/home/data.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 200704, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_data1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/data.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/data1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "type": "unknown"}]}
4. Do dd in guest
(guest)# dd if=/dev/urandom of=test bs=1M count=1000
5. Query block info
{"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel820-64-virtio.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5894922240, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel820-64-virtio.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/image1/virtio-backend", "type": "unknown"}, {"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 2147483648, "filename": "/home/data.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 1135878144, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_data1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1150877696}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/data.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/data1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1150877696}], "type": "unknown"}]}
6. Shutdown vm
(qemu)system_powerdown
7. Check bitmap info in image
# qemu-img info /home/data.qcow2
image: /home/data.qcow2
file format: qcow2
virtual size: 2 GiB (2147483648 bytes)
disk size: 1.06 GiB
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
bitmaps:
[0]:
flags:
[0]: auto
name: bitmap0
granularity: 65536
refcount bits: 16
corrupt: false
|
Description of problem: Qemu core dump when add a persistent bitmap Version-Release number of selected component (if applicable): kernel version:4.18.0-147.el8.x86_64 qemu-kvm version:qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64 How reproducible: 100% Steps to Reproduce: 1.Start guest with qemu cmds: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine q35 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x1 \ -m 7168 \ -smp 4,maxcpus=4,cores=2,threads=1,dies=1,sockets=2 \ -cpu 'Skylake-Client',+kvm_pv_unhalt \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20191118-011823-gEG3j1ms,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20191118-011823-gEG3j1mt,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=id4p8G4l \ -chardev socket,server,id=chardev_serial0,path=/var/tmp/serial-serial0-20191118-011823-gEG3j1mt,nowait \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20191118-011823-gEG3j1mt,path=/var/tmp/seabios-20191118-011823-gEG3j1mt,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20191118-011823-gEG3j1mt,iobase=0x402 \ -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \ -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \ -object iothread,id=iothread0 \ -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel820-64-virtio.qcow2 \ -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pcie.0-root-port-3,addr=0x0,iothread=iothread0 \ -drive id=drive_data1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/aliang/data.qcow2 \ -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \ -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \ -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \ -device virtio-net-pci,mac=9a:4f:f4:e5:bd:67,id=idkQvhgf,netdev=idnMcj5J,bus=pcie.0-root-port-4,addr=0x0 \ -netdev tap,id=idnMcj5J,vhost=on \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ -device virtio-serial-pci,id=virtio-serial0,bus=pcie.0,addr=0x8 \ -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \ -device virtserialport,bus=virtio-serial0.0,chardev=qga0,id=qemu-ga0,name=org.qemu.guest_agent.0 \ 2. Add persistent bitmap to data disk { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_data1", "name": "bitmap0","persistent":true}} Ncat: Connection reset by peer. Actual results: After step2, qemu core dump with info: (qemu) qemu: qemu_mutex_unlock_impl: Operation not permitted test.txt: line 41: 29879 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -machine q35 -nodefaults -device VGA,bus=pcie.0,addr=0x1 -m 7168 -smp 4,maxcpus=4,cores=2,threads=1,dies=1,sockets=2 -cpu 'Skylake-Client',+kvm_pv_unhalt ... gdb info: (gdb) bt #0 0x00007feaadca48df in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007feaadc8ecf5 in __GI_abort () at abort.c:79 #2 0x00005646b76058ce in error_exit (err=<optimized out>, msg=msg@entry=0x5646b77ab6d0 <__func__.16350> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36 #3 0x00005646b7605bba in qemu_mutex_unlock_impl (mutex=mutex@entry=0x5646b8b75f90, file=file@entry=0x5646b77aaa7f "util/async.c", line=line@entry=526) at util/qemu-thread-posix.c:108 #4 0x00005646b7600b29 in aio_context_release (ctx=ctx@entry=0x5646b8b75f30) at util/async.c:526 #5 0x00005646b758dc58 in bdrv_can_store_new_dirty_bitmap (bs=bs@entry=0x5646b8b9e4b0, name=name@entry=0x5646b92eff30 "bitmap0", granularity=granularity@entry=65536, errp=errp@entry=0x7ffde1df7c68) at block/dirty-bitmap.c:536 #6 0x00005646b73dacd3 in qmp_block_dirty_bitmap_add (errp=0x7ffde1df7c68, disabled=false, has_disabled=<optimized out>, persistent=<optimized out>, has_persistent=true, granularity=65536, has_granularity=<optimized out>, name=0x5646b92eff30 "bitmap0", node=<optimized out>) at blockdev.c:2894 #7 0x00005646b73dacd3 in qmp_block_dirty_bitmap_add (node=<optimized out>, name=0x5646b92eff30 "bitmap0", has_granularity=<optimized out>, granularity=<optimized out>, has_persistent=true, persistent=<optimized out>, has_disabled=false, disabled=false, errp=0x7ffde1df7c68) at blockdev.c:2856 #8 0x00005646b74f5833 in qmp_marshal_block_dirty_bitmap_add (args=<optimized out>, ret=<optimized out>, errp=0x7ffde1df7ce8) at qapi/qapi-commands-block-core.c:651 #9 0x00005646b75b898c in do_qmp_dispatch (errp=0x7ffde1df7ce0, allow_oob=<optimized out>, request=<optimized out>, cmds=0x5646b7ea4cc0 <qmp_commands>) at qapi/qmp-dispatch.c:132 #10 0x00005646b75b898c in qmp_dispatch (cmds=0x5646b7ea4cc0 <qmp_commands>, request=<optimized out>, allow_oob=<optimized out>) at qapi/qmp-dispatch.c:175 #11 0x00005646b74d7141 in monitor_qmp_dispatch (mon=0x5646b8ba2a30, req=<optimized out>) at monitor/qmp.c:120 #12 0x00005646b74d778a in monitor_qmp_bh_dispatcher (data=<optimized out>) at monitor/qmp.c:209 #13 0x00005646b7600366 in aio_bh_call (bh=0x5646b8ad6c60) at util/async.c:117 #14 0x00005646b7600366 in aio_bh_poll (ctx=ctx@entry=0x5646b8ad5840) at util/async.c:117 #15 0x00005646b7603754 in aio_dispatch (ctx=0x5646b8ad5840) at util/aio-posix.c:459 #16 0x00005646b7600242 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:260 --Type <RET> for more, q to quit, c to continue without paging-- #17 0x00007feab252667d in g_main_dispatch (context=0x5646b8b618b0) at gmain.c:3176 #18 0x00007feab252667d in g_main_context_dispatch (context=context@entry=0x5646b8b618b0) at gmain.c:3829 #19 0x00005646b7602808 in glib_pollfds_poll () at util/main-loop.c:219 #20 0x00005646b7602808 in os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:242 #21 0x00005646b7602808 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:518 #22 0x00005646b73e4201 in main_loop () at vl.c:1828 #23 0x00005646b7290b82 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4504 Expected results: Add persistent bitmap can executed successfully. Additional info: Will attach coredump info later.