Bug 1744955 - Qemu hang when block resize a qcow2 image
Summary: Qemu hang when block resize a qcow2 image
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.1
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: ---
Assignee: Sergio Lopez
QA Contact: aihua liang
URL:
Whiteboard:
: 1716316 (view as bug list)
Depends On:
Blocks: 1720182
TreeView+ depends on / blocked
 
Reported: 2019-08-23 11:00 UTC by aihua liang
Modified: 2019-11-06 07:19 UTC (History)
9 users (show)

Fixed In Version: qemu-kvm-4.1.0-13.module+el8.1.0+4313+ef76ec61
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-06 07:19:01 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 0 None None None 2019-11-06 07:19:24 UTC

Description aihua liang 2019-08-23 11:00:27 UTC 
Description of problem:
 Qemu hang when block resize a qcow2 image with persistent bitmap stored on it

Version-Release number of selected component (if applicable):
 kernel version:4.18.0-134.el8.x86_64
 qemu-kvm version:qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64

How reproducible:
 100%

Steps to Reproduce:
1.Start guest with 2G data disk that stored on gluster server
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190820-032540-OesJUJdj,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190820-032540-OesJUJdj,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idbJPqrG \
    -chardev socket,id=chardev_serial0,server,path=/var/tmp/serial-serial0-20190820-032540-OesJUJdj,nowait \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20190820-032540-OesJUJdj,path=/var/tmp/seabios-20190820-032540-OesJUJdj,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190820-032540-OesJUJdj,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -drive id=drive_image1,if=none,snapshot=off,cache=none,format=qcow2,file=/mnt/nfs/rhel810-64-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pcie.0-root-port-3,addr=0x0,iothread=iothread0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -drive id=drive_data1,if=none,snapshot=off,cache=none,format=qcow2,file=/mnt/nfs/data.qcow2 \
    -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread1 \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:19:6a:3c:a6:a5,id=idq14C2Q,netdev=idHzG7Zk,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idHzG7Zk,vhost=on \
    -m 7168  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -monitor stdio \
    -device virtio-serial-pci,id=virtio-serial0,bus=pcie_extra_root_port_0,addr=0x0 \
    -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \
    -device virtserialport,bus=virtio-serial0.0,chardev=qga0,id=qemu-ga0,name=org.qemu.guest_agent.0 \
    -qmp tcp:0:3000,server,nowait \

2. Add persistent bitmap on data disk, then shutdown vm
    { 'execute': 'block-dirty-bitmap-add', 'arguments': {'node':'drive_data1','name':'bitmap','persistent':true}}
  (qemu)quit

3. Start vm again, then resize data disk to 3G.
     { "execute": "block_resize", "arguments": { "device": "drive_data1", "size":3221225472}} 

Actual results:
After step3, qemu hang
(gdb) bt
#0  0x00007f121a5cb8dd in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007f121a5c4bc4 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x55cedca88460)
    at ../nptl/pthread_mutex_lock.c:115
#2  0x000055cedb29559d in qemu_mutex_lock_impl
    (mutex=0x55cedca88460, file=0x55cedb437fdf "util/async.c", line=510) at util/qemu-thread-posix.c:66
#3  0x000055cedb217112 in bdrv_truncate
    (child=<optimized out>, offset=offset@entry=3221225472, prealloc=prealloc@entry=PREALLOC_MODE_OFF, errp=errp@entry=0x7ffd25c46838) at block/io.c:3254
#4  0x000055cedb20353f in blk_truncate
    (blk=blk@entry=0x55cedd0f9330, offset=offset@entry=3221225472, prealloc=prealloc@entry=PREALLOC_MODE_OFF, errp=errp@entry=0x7ffd25c46838) at block/block-backend.c:2041
#5  0x000055cedb072411 in qmp_block_resize
    (has_device=<optimized out>, device=0x55cedca9e170 "drive_data1", has_node_name=<optimized out>, node_name=<optimized out>, size=3221225472, errp=errp@entry=0x7ffd25c46838) at blockdev.c:3163
#6  0x000055cedb188e27 in qmp_marshal_block_resize
    (args=<optimized out>, ret=<optimized out>, errp=0x7ffd25c468b8)
    at qapi/qapi-commands-block-core.c:279
#7  0x000055cedb248ecc in do_qmp_dispatch
    (errp=0x7ffd25c468b0, allow_oob=<optimized out>, request=<optimized out>, cmds=0x55cedbb247a0 <qmp_commands>) at qapi/qmp-dispatch.c:131
#8  0x000055cedb248ecc in qmp_dispatch
    (cmds=0x55cedbb247a0 <qmp_commands>, request=<optimized out>, allow_oob=<optimized out>)
    at qapi/qmp-dispatch.c:174
#9  0x000055cedb16b4f1 in monitor_qmp_dispatch (mon=0x55cedcab3600, req=<optimized out>)
    at monitor/qmp.c:120
#10 0x000055cedb16bb3a in monitor_qmp_bh_dispatcher (data=<optimized out>) at monitor/qmp.c:209
--Type <RET> for more, q to quit, c to continue without paging--  
#11 0x000055cedb28fc26 in aio_bh_call (bh=0x55cedc9efb20) at util/async.c:117
#12 0x000055cedb28fc26 in aio_bh_poll (ctx=ctx@entry=0x55cedc9ee6d0) at util/async.c:117
#13 0x000055cedb293064 in aio_dispatch (ctx=0x55cedc9ee6d0) at util/aio-posix.c:459
#14 0x000055cedb28fb02 in aio_ctx_dispatch
    (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:260
#15 0x00007f121f0ac67d in g_main_dispatch (context=0x55cedca77e70) at gmain.c:3176
#16 0x00007f121f0ac67d in g_main_context_dispatch (context=context@entry=0x55cedca77e70) at gmain.c:3829
#17 0x000055cedb292118 in glib_pollfds_poll () at util/main-loop.c:218
#18 0x000055cedb292118 in os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:241
#19 0x000055cedb292118 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:517
#20 0x000055cedb07b169 in main_loop () at vl.c:1809
#21 0x000055cedaf2afd3 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4506

Expected results:
 block_resize executed successfully.

Additional info:
 pstack info:
  pstack 12723
Thread 10 (Thread 0x7f11f8dff700 (LWP 12779)):
#0  0x00007f121a5c847c in futex_wait_cancelable (private=0, expected=0, futex_word=0x55cedd6a7c78) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  0x00007f121a5c847c in __pthread_cond_wait_common (abstime=0x0, mutex=0x55cedd6a7c88, cond=0x55cedd6a7c50) at pthread_cond_wait.c:502
#2  0x00007f121a5c847c in __pthread_cond_wait (cond=0x55cedd6a7c50, mutex=mutex@entry=0x55cedd6a7c88) at pthread_cond_wait.c:655
#3  0x000055cedb29586d in qemu_cond_wait_impl (cond=<optimized out>, mutex=0x55cedd6a7c88, file=0x55cedb411c37 "ui/vnc-jobs.c", line=214) at util/qemu-thread-posix.c:161
#4  0x000055cedb1bed71 in vnc_worker_thread_loop (queue=queue@entry=0x55cedd6a7c50) at ui/vnc-jobs.c:214
#5  0x000055cedb1bf330 in vnc_worker_thread (arg=0x55cedd6a7c50) at ui/vnc-jobs.c:324
#6  0x000055cedb2954b4 in qemu_thread_start (args=0x55cedd0f98c0) at util/qemu-thread-posix.c:502
#7  0x00007f121a5c22de in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007f121a2f3133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 9 (Thread 0x7f11faffd700 (LWP 12740)):
#0  0x00007f121a5cb8dd in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007f121a5c4af9 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x55cedbaf1f60 <qemu_global_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2  0x000055cedb29559d in qemu_mutex_lock_impl (mutex=0x55cedbaf1f60 <qemu_global_mutex>, file=0x55cedb33aa58 "/builddir/build/BUILD/qemu-4.1.0/accel/kvm/kvm-all.c", line=2353) at util/qemu-thread-posix.c:66
#3  0x000055cedaf7639e in qemu_mutex_lock_iothread_impl (file=file@entry=0x55cedb33aa58 "/builddir/build/BUILD/qemu-4.1.0/accel/kvm/kvm-all.c", line=line@entry=2353) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1859
#4  0x000055cedaf91408 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/accel/kvm/kvm-all.c:2353
#5  0x000055cedaf7656e in qemu_kvm_cpu_thread_fn (arg=0x55cedcb883a0) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1285
#6  0x000055cedb2954b4 in qemu_thread_start (args=0x55cedcbab120) at util/qemu-thread-posix.c:502
#7  0x00007f121a5c22de in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007f121a2f3133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 8 (Thread 0x7f11fb7fe700 (LWP 12739)):
#0  0x00007f121a5cb8dd in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007f121a5c4af9 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x55cedbaf1f60 <qemu_global_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2  0x000055cedb29559d in qemu_mutex_lock_impl (mutex=0x55cedbaf1f60 <qemu_global_mutex>, file=0x55cedb32f068 "/builddir/build/BUILD/qemu-4.1.0/exec.c", line=3301) at util/qemu-thread-posix.c:66
#3  0x000055cedaf7639e in qemu_mutex_lock_iothread_impl (file=<optimized out>, line=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1859
#4  0x000055cedaf2e8f9 in prepare_mmio_access (mr=<optimized out>, mr=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3301
#5  0x000055cedaf2f990 in flatview_write_continue (fv=0x7f11e8628a20, addr=4271965456, attrs=..., buf=0x7f121f9f2028 "", len=2, addr1=<optimized out>, l=<optimized out>, mr=0x55cedd47ff90) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3332
#6  0x000055cedaf2fb46 in flatview_write (fv=0x7f11e8628a20, addr=4271965456, attrs=..., buf=0x7f121f9f2028 "", len=2) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3376
#7  0x000055cedaf33d6f in address_space_write (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3466
#8  0x000055cedaf914ca in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/accel/kvm/kvm-all.c:2298
#9  0x000055cedaf7656e in qemu_kvm_cpu_thread_fn (arg=0x55cedcb64b80) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1285
#10 0x000055cedb2954b4 in qemu_thread_start (args=0x55cedcb87b60) at util/qemu-thread-posix.c:502
#11 0x00007f121a5c22de in start_thread (arg=<optimized out>) at pthread_create.c:486
#12 0x00007f121a2f3133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 7 (Thread 0x7f11fbfff700 (LWP 12738)):
#0  0x00007f121a5cb8dd in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007f121a5c4af9 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x55cedbaf1f60 <qemu_global_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2  0x000055cedb29559d in qemu_mutex_lock_impl (mutex=0x55cedbaf1f60 <qemu_global_mutex>, file=0x55cedb32f068 "/builddir/build/BUILD/qemu-4.1.0/exec.c", line=3301) at util/qemu-thread-posix.c:66
#3  0x000055cedaf7639e in qemu_mutex_lock_iothread_impl (file=<optimized out>, line=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1859
#4  0x000055cedaf2e8f9 in prepare_mmio_access (mr=<optimized out>, mr=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3301
#5  0x000055cedaf3397f in flatview_read_continue (fv=0x7f11e872a800, addr=1017, attrs=..., buf=<optimized out>, len=1, addr1=<optimized out>, l=<optimized out>, mr=0x55cedd521300) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3396
#6  0x000055cedaf33ba3 in flatview_read (fv=0x7f11e872a800, addr=1017, attrs=..., buf=0x7f121f9f6000 "", len=1) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3436
#7  0x000055cedaf33ccf in address_space_read_full (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3449
#8  0x000055cedaf91544 in kvm_handle_io (count=1, size=1, direction=<optimized out>, data=<optimized out>, attrs=..., port=1017) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/accel/kvm/kvm-all.c:2042
#9  0x000055cedaf91544 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/accel/kvm/kvm-all.c:2288
#10 0x000055cedaf7656e in qemu_kvm_cpu_thread_fn (arg=0x55cedcb40a50) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1285
#11 0x000055cedb2954b4 in qemu_thread_start (args=0x55cedcb64340) at util/qemu-thread-posix.c:502
#12 0x00007f121a5c22de in start_thread (arg=<optimized out>) at pthread_create.c:486
#13 0x00007f121a2f3133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 6 (Thread 0x7f1210caa700 (LWP 12737)):
#0  0x00007f121a5cb8dd in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007f121a5c4af9 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x55cedbaf1f60 <qemu_global_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2  0x000055cedb29559d in qemu_mutex_lock_impl (mutex=0x55cedbaf1f60 <qemu_global_mutex>, file=0x55cedb33aa58 "/builddir/build/BUILD/qemu-4.1.0/accel/kvm/kvm-all.c", line=2353) at util/qemu-thread-posix.c:66
#3  0x000055cedaf7639e in qemu_mutex_lock_iothread_impl (file=file@entry=0x55cedb33aa58 "/builddir/build/BUILD/qemu-4.1.0/accel/kvm/kvm-all.c", line=line@entry=2353) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1859
#4  0x000055cedaf91408 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/accel/kvm/kvm-all.c:2353
#5  0x000055cedaf7656e in qemu_kvm_cpu_thread_fn (arg=0x55cedcaf2520) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1285
#6  0x000055cedb2954b4 in qemu_thread_start (args=0x55cedcb155e0) at util/qemu-thread-posix.c:502
#7  0x00007f121a5c22de in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007f121a2f3133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 5 (Thread 0x7f12114ab700 (LWP 12736)):
#0  0x00007f121a2e8211 in __GI___poll (fds=0x55cedcab5040, nfds=5, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f121f0ac9b6 in g_main_context_poll (priority=<optimized out>, n_fds=5, fds=0x55cedcab5040, timeout=<optimized out>, context=0x55cedcab3bc0) at gmain.c:4203
#2  0x00007f121f0ac9b6 in g_main_context_iterate (context=0x55cedcab3bc0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3897
#3  0x00007f121f0acd72 in g_main_loop_run (loop=0x55cedcaa31f0) at gmain.c:4098
#4  0x000055cedb075b31 in iothread_run (opaque=0x55cedca18860) at iothread.c:82
#5  0x000055cedb2954b4 in qemu_thread_start (args=0x55cedcab3d20) at util/qemu-thread-posix.c:502
#6  0x00007f121a5c22de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f121a2f3133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 4 (Thread 0x7f1212872700 (LWP 12726)):
#0  0x00007f121a5cb8dd in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007f121a5c4af9 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x55cedbaf1f60 <qemu_global_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2  0x000055cedb29559d in qemu_mutex_lock_impl (mutex=0x55cedbaf1f60 <qemu_global_mutex>, file=0x55cedb32f068 "/builddir/build/BUILD/qemu-4.1.0/exec.c", line=3301) at util/qemu-thread-posix.c:66
#3  0x000055cedaf7639e in qemu_mutex_lock_iothread_impl (file=<optimized out>, line=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/cpus.c:1859
#4  0x000055cedaf2e8f9 in prepare_mmio_access (mr=<optimized out>, mr=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/exec.c:3301
#5  0x000055cedaf362f4 in address_space_stl_le (as=<optimized out>, addr=<optimized out>, val=16424, attrs=..., result=0x0) at /usr/src/debug/qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64/memory_ldst.inc.c:338
#6  0x000055cedb216e4a in bdrv_parent_cb_resize (bs=0x55cedcab5450) at block/io.c:3125
#7  0x000055cedb216e4a in bdrv_co_write_req_finish (child=<optimized out>, req=0x7f1212071f00, req=0x7f1212071f00, ret=0, bytes=<optimized out>, offset=3221225472) at block/io.c:1667
#8  0x000055cedb216e4a in bdrv_co_truncate (child=<optimized out>, offset=<optimized out>, prealloc=PREALLOC_MODE_OFF, errp=<optimized out>) at block/io.c:3211
#9  0x000055cedb21700b in bdrv_truncate_co_entry (opaque=0x7ffd25c46790) at block/io.c:3231
#10 0x000055cedb2a8b53 in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at util/coroutine-ucontext.c:115
#11 0x00007f121a2444b0 in __start_context () at ../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91
#12 0x00007ffd25c45fc0 in  ()
#13 0x0000000000000000 in  ()
Thread 3 (Thread 0x7f1213073700 (LWP 12725)):
#0  0x00007f121a2e8306 in __GI_ppoll (fds=0x7f1204001fb0, nfds=2, timeout=<optimized out>, timeout@entry=0x7f1213072640, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:39
#1  0x000055cedb291175 in ppoll (__ss=0x0, __timeout=0x7f1213072640, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2  0x000055cedb291175 in qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at util/qemu-timer.c:334
#3  0x000055cedb2931d4 in aio_poll (ctx=0x55cedca87a20, blocking=blocking@entry=true) at util/aio-posix.c:669
#4  0x000055cedb075b04 in iothread_run (opaque=0x55cedca6e760) at iothread.c:75
#5  0x000055cedb2954b4 in qemu_thread_start (args=0x55cedca87f30) at util/qemu-thread-posix.c:502
#6  0x00007f121a5c22de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f121a2f3133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 2 (Thread 0x7f1213874700 (LWP 12724)):
#0  0x00007f121a2ed99d in syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055cedb295cdf in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at util/qemu-thread-posix.c:438
#2  0x000055cedb295cdf in qemu_event_wait (ev=ev@entry=0x55cedbb26ec8 <rcu_call_ready_event>) at util/qemu-thread-posix.c:442
#3  0x000055cedb2a7862 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:260
#4  0x000055cedb2954b4 in qemu_thread_start (args=0x55cedc97f5a0) at util/qemu-thread-posix.c:502
#5  0x00007f121a5c22de in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f121a2f3133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 1 (Thread 0x7f121f9c2ec0 (LWP 12723)):
#0  0x00007f121a5cb8dd in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007f121a5c4bc4 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x55cedca88460) at ../nptl/pthread_mutex_lock.c:115
#2  0x000055cedb29559d in qemu_mutex_lock_impl (mutex=0x55cedca88460, file=0x55cedb437fdf "util/async.c", line=510) at util/qemu-thread-posix.c:66
#3  0x000055cedb217112 in bdrv_truncate (child=<optimized out>, offset=offset@entry=3221225472, prealloc=prealloc@entry=PREALLOC_MODE_OFF, errp=errp@entry=0x7ffd25c46838) at block/io.c:3254
#4  0x000055cedb20353f in blk_truncate (blk=blk@entry=0x55cedd0f9330, offset=offset@entry=3221225472, prealloc=prealloc@entry=PREALLOC_MODE_OFF, errp=errp@entry=0x7ffd25c46838) at block/block-backend.c:2041
#5  0x000055cedb072411 in qmp_block_resize (has_device=<optimized out>, device=0x55cedca9e170 "drive_data1", has_node_name=<optimized out>, node_name=<optimized out>, size=3221225472, errp=errp@entry=0x7ffd25c46838) at blockdev.c:3163
#6  0x000055cedb188e27 in qmp_marshal_block_resize (args=<optimized out>, ret=<optimized out>, errp=0x7ffd25c468b8) at qapi/qapi-commands-block-core.c:279
#7  0x000055cedb248ecc in do_qmp_dispatch (errp=0x7ffd25c468b0, allow_oob=<optimized out>, request=<optimized out>, cmds=0x55cedbb247a0 <qmp_commands>) at qapi/qmp-dispatch.c:131
#8  0x000055cedb248ecc in qmp_dispatch (cmds=0x55cedbb247a0 <qmp_commands>, request=<optimized out>, allow_oob=<optimized out>) at qapi/qmp-dispatch.c:174
#9  0x000055cedb16b4f1 in monitor_qmp_dispatch (mon=0x55cedcab3600, req=<optimized out>) at monitor/qmp.c:120
#10 0x000055cedb16bb3a in monitor_qmp_bh_dispatcher (data=<optimized out>) at monitor/qmp.c:209
#11 0x000055cedb28fc26 in aio_bh_call (bh=0x55cedc9efb20) at util/async.c:117
#12 0x000055cedb28fc26 in aio_bh_poll (ctx=ctx@entry=0x55cedc9ee6d0) at util/async.c:117
#13 0x000055cedb293064 in aio_dispatch (ctx=0x55cedc9ee6d0) at util/aio-posix.c:459
#14 0x000055cedb28fb02 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:260
#15 0x00007f121f0ac67d in g_main_dispatch (context=0x55cedca77e70) at gmain.c:3176
#16 0x00007f121f0ac67d in g_main_context_dispatch (context=context@entry=0x55cedca77e70) at gmain.c:3829
#17 0x000055cedb292118 in glib_pollfds_poll () at util/main-loop.c:218
#18 0x000055cedb292118 in os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:241
#19 0x000055cedb292118 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:517
#20 0x000055cedb07b169 in main_loop () at vl.c:1809
#21 0x000055cedaf2afd3 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4506
Comment 2 Ademar Reis 2019-08-23 18:51:40 UTC 
We can fix this kind of bug in 8.1.1, so setting ITR.
Comment 4 aihua liang 2019-08-26 02:31:17 UTC 
Only virtio_blk + dataplane hit this issue.
Comment 5 Sergio Lopez 2019-09-12 18:34:35 UTC 
Resizing a virtio-blk device running on an iothread triggers a deadlock between this one and the main thread. This is still reproducible in the latest upstream. I've sent an RFC patch to discuss a possible solution:

 - https://lists.gnu.org/archive/html/qemu-block/2019-09/msg00564.html
Comment 6 John Ferlan 2019-09-17 14:06:20 UTC 
Updated the title to reflect the problem is more generic than for bitmaps. 

Moved to ASSIGNED since patch is posted (at v3 now: https://lists.nongnu.org/archive/html/qemu-devel/2019-09/msg03201.html)
Comment 11 CongLi 2019-09-27 07:58:20 UTC 
*** Bug 1716316 has been marked as a duplicate of this bug. ***
Comment 17 aihua liang 2019-09-29 02:55:22 UTC 
Verified on qemu-kvm-4.1.0-13.module+el8.1.0+4313+ef76ec61.x86_64, the issue has been resolved, set bug's status to "Verified".

Test Steps:
  1.Start guest with 2G data disk.
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190820-032540-OesJUJdj,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190820-032540-OesJUJdj,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idbJPqrG \
    -chardev socket,id=chardev_serial0,server,path=/var/tmp/serial-serial0-20190820-032540-OesJUJdj,nowait \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20190820-032540-OesJUJdj,path=/var/tmp/seabios-20190820-032540-OesJUJdj,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190820-032540-OesJUJdj,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -drive id=drive_image1,if=none,snapshot=off,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel810-64-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pcie.0-root-port-3,addr=0x0,iothread=iothread0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -drive id=drive_data1,if=none,snapshot=off,cache=none,format=qcow2,file=/home/data.qcow2 \
    -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread1 \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:19:6a:3c:a6:a5,id=idq14C2Q,netdev=idHzG7Zk,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idHzG7Zk,vhost=on \
    -m 7168  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -monitor stdio \
    -device virtio-serial-pci,id=virtio-serial0,bus=pcie_extra_root_port_0,addr=0x0 \
    -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \
    -device virtserialport,bus=virtio-serial0.0,chardev=qga0,id=qemu-ga0,name=org.qemu.guest_agent.0 \
    -qmp tcp:0:3000,server,nowait \

2. Add persistent bitmap on data disk, then shutdown vm
    { 'execute': 'block-dirty-bitmap-add', 'arguments': {'node':'drive_data1','name':'bitmap','persistent':true}}

    {"execute":"query-block"}
    {"io-status": "ok", "device": "drive_data1", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 2147483648, "filename": "/home/data.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 69279744, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "#block363", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/data.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/data1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "type": "unknown"}

    {"execute":"system_powerdown"}
    {"return": {}}
{"timestamp": {"seconds": 1569724436, "microseconds": 965799}, "event": "POWERDOWN"}
{"timestamp": {"seconds": 1569724587, "microseconds": 517683}, "event": "SHUTDOWN", "data": {"guest": true, "reason": "guest-shutdown"}}
    

3. Start vm again, then resize data disk to 3G.
     { "execute": "block_resize", "arguments": { "device": "drive_data1", "size":3221225472}}
     {"return": {}}

4. Check image size online.
     #qemu-img info /home/data.qcow2 -U
     image: /home/data.qcow2
file format: qcow2
virtual size: 3 GiB (3221225472 bytes)
disk size: 66.1 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

5. Shutdown vm, resize it to 4G
   #qemu-img resize /home/data.qcow2 +1G
    Image resized.
   #qemu-img info /home/data.qcow2
   image: /home/data.qcow2
file format: qcow2
virtual size: 4 GiB (4294967296 bytes)
disk size: 166 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: bitmap
            granularity: 65536
    refcount bits: 16
    corrupt: false

6. Restart vm again, check block info, then resize data disk to 3G, shutdown vm
   {"execute":"query-block"}
   {"io-status": "ok", "device": "drive_data1", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 4294967296, "filename": "/home/data.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 174407680, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "#block364", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/data.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/data1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 105316352}], "type": "unknown"}

  { "execute": "block_resize", "arguments": { "device": "drive_data1", "size":3221225472}}
{"return": {}}

  {"execute":"query-block"}
  {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 3221225472, "filename": "/home/data.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 174403584, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "#block364", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/data.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/data1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 105316352}], "type": "unknown"}

  {"execute":"system_powerdown"}
{"return": {}}
{"timestamp": {"seconds": 1569724436, "microseconds": 965799}, "event": "POWERDOWN"}
{"timestamp": {"seconds": 1569724587, "microseconds": 517683}, "event": "SHUTDOWN", "data": {"guest": true, "reason": "guest-shutdown"}}

7. Check image info offline
  #qemu-img info /home/data.qcow2
image: /home/data.qcow2
file format: qcow2
virtual size: 3 GiB (3221225472 bytes)
disk size: 166 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: bitmap
            granularity: 65536
    refcount bits: 16
    corrupt: false
     
8. Resize data disk to 2G, and check image info
   #qemu-img resize --shrink data.qcow2 -1G
Image resized.

   #qemu-img info /home/data.qcow2
image: /home/data.qcow2
file format: qcow2
virtual size: 2 GiB (2147483648 bytes)
disk size: 166 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: bitmap
            granularity: 65536
    refcount bits: 16
    corrupt: false

8. Restart vm again, check data disk info:
   {"execute":"query-block"}
   {"io-status": "ok", "device": "drive_data1", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 2147483648, "filename": "/home/data.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 174407680, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "#block318", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/data.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/data1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 105316352}], "type": "unknown"}

9. Shutdown vm
   {"execute":"system_powerdown"}
Comment 19 errata-xmlrpc 2019-11-06 07:19:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723
Note You need to log in before you can comment on or make changes to this bug.