+++ This bug was initially created as a clone of Bug #1716316 +++

Description of problem:
Guest will hang after 'blockdec_resize' a disk by hotplugging with iothread. And without iothread, there is no the issue.


Version-Release number of selected component (if applicable):
kernel: 4.18.0-91.el8.x86_64
qemu: qemu-kvm-4.0.0-2.module+el8.1.0+3258+4c45705b


How reproducible:
100%


Steps to Reproduce:
1. Boot guest from a win10 image.

Full command lines:
# /usr/libexec/qemu-kvm \
    -name 'guest-win2019' \
    -machine q35 \
    -nodefaults \
    -vga qxl \
    -object iothread,id=iothread0 \
    -drive file=/home/test/win10/win10-64-virtio.qcow2,if=none,format=qcow2,rerror=stop,werror=stop,cache=none,id=drive_image1 \
    -device  pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device  virtio-blk-pci,drive=drive_image1,write-cache=on,iothread=iothread0,bus=pcie.0-root-port-3 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -vnc :0  \
    -monitor stdio \
    -m 8192 \
    -smp 8 \
    -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
    -netdev tap,id=idxgXAlm \
    -qmp tcp:localhost:5902,server,nowait  \
    -device nec-usb-xhci,id=usb1,bus=pcie.0,addr=0x4 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \

2.Create a data image.
# qemu-img create -f qcow2 /home/test/win10/data.qcow2 10G

3.Hotplug a blk device in QMP: 
{'execute':'qmp_capabilities'}

{"execute":"blockdev-add","arguments":{"node-name":"file_image3","driver":"qcow2","file":{"driver":"file","filename":"/home/test/win10/data.qcow2"}}}
{"return": {}}

{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg0", "drive": "file_image3", "iothread": "iothread0", "bus": "pcie.0-root-port-5"}, "id": "test"}
{"return": {}, "id": "test"}

4. block_resize the blk device
{"execute":"block_resize","arguments":{"node-name":"file_image3","size":21474836480 }} ---------- No return!


Actual results:
After blockdev_resize, no {"return": {}}, and the guest hangs.


Expected results:
After blockdev_resize, return value and guest works normally.


Additional info:
1. Linux guest also hits this issue.
2. If boot guest without iothread, there is no the issue any more.

--- Additional comment from yujie ma on 2019-06-03 09:30:16 UTC ---

strace qemu-kvm process log :

...
ppoll([{fd=0, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=15, events=POLLIN}, {fd=17, events=POLLIN}, {fd=29, events=POLLIN}, {fd=35, events=POLLIN}, {fd=38, events=POLLIN}, {fd=39, events=POLLIN}, {fd=40, events=POLLIN}], 12, {tv_sec=0, tv_nsec=304350}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {tv_sec=0, tv_nsec=198639})
read(4, "\1\0\0\0\0\0\0\0", 512)        = 8
rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0
mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7cb80cd000
mprotect(0x7f7cb80cd000, 4096, PROT_NONE) = 0
rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM IO], 8) = 0
write(10, "\1\0\0\0\0\0\0\0", 8)        = 8
ppoll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, NULL, NULL, 8) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\2\0\0\0\0\0\0\0", 512)        = 8
ppoll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, NULL, NULL, 8) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\2\0\0\0\0\0\0\0", 512)        = 8
ppoll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, NULL, NULL, 8) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\2\0\0\0\0\0\0\0", 512)        = 8
ppoll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, NULL, NULL, 8) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\2\0\0\0\0\0\0\0", 512)        = 8
ppoll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, NULL, NULL, 8) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\4\0\0\0\0\0\0\0", 512)        = 8
ppoll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, NULL, NULL, 8) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\6\0\0\0\0\0\0\0", 512)        = 8
ppoll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, NULL, NULL, 8) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\2\0\0\0\0\0\0\0", 512)        = 8
futex(0x560a08587e20, FUTEX_WAIT_PRIVATE, 2, NULL

--- Additional comment from yujie ma on 2019-06-03 10:22:44 UTC ---

Not hit this issue on:

qemu-kvm-2.12.0-75.module+el8.1.0+3252+aa5f0857
qemu-kvm-3.1.0-27.module+el8.0.1+3253+c5371cb

So it is a regression issue.

--- Additional comment from Kevin Wolf on 2019-06-04 08:30:48 UTC ---

I could reproduce this after a few failed attempts. The important part is that the guest OS has had time to initialise the virtio-blk driver for the new device before resizing, so that the device is actually running in the iothread.

The problem is that BlockDevOps.resize_cb is called from the iothread, but the virtio-blk .resize_cb handler takes the BQL, which it must not do from an iothread. This causes a deadlock between the main thread and the iothread:

Thread 3 (Thread 0x7f6dee5ce700 (LWP 12621)):
#0  0x00007f6df99a9b8d in __lll_lock_wait () at /lib64/libpthread.so.0
#1  0x00007f6df99a2da9 in pthread_mutex_lock () at /lib64/libpthread.so.0
#2  0x000055a8be92b213 in qemu_mutex_lock_impl (mutex=0x55a8bf123380 <qemu_global_mutex>, file=0x55a8bea36288 "/home/kwolf/source/qemu/exec.c", line=3284) at util/qemu-thread-posix.c:66
#3  0x000055a8be36c4d6 in qemu_mutex_lock_iothread_impl (file=0x55a8bea36288 "/home/kwolf/source/qemu/exec.c", line=3284) at /home/kwolf/source/qemu/cpus.c:1859
#4  0x000055a8be31e529 in prepare_mmio_access (mr=0x55a8bfb9f500) at /home/kwolf/source/qemu/exec.c:3284
#5  0x000055a8be31febc in address_space_stl_internal (as=0x55a8c03f2380, addr=4276092928, val=16420, attrs=..., result=0x0, endian=DEVICE_LITTLE_ENDIAN) at /home/kwolf/source/qemu/memory_ldst.inc.c:338
#6  0x000055a8be320032 in address_space_stl_le (as=0x55a8c03f2380, addr=4276092928, val=16420, attrs=..., result=0x0) at /home/kwolf/source/qemu/memory_ldst.inc.c:386
#7  0x000055a8be681eea in msi_send_message (dev=0x55a8c03f2170, msg=...) at hw/pci/msi.c:340
#8  0x000055a8be680a73 in msix_notify (dev=0x55a8c03f2170, vector=0) at hw/pci/msix.c:492
#9  0x000055a8be726de4 in virtio_pci_notify (d=0x55a8c03f2170, vector=0) at hw/virtio/virtio-pci.c:67
#10 0x000055a8be430a62 in virtio_notify_vector (vdev=0x55a8c03fa2f0, vector=0) at /home/kwolf/source/qemu/hw/virtio/virtio.c:1125
#11 0x000055a8be432427 in virtio_notify_config (vdev=0x55a8c03fa2f0) at /home/kwolf/source/qemu/hw/virtio/virtio.c:1740
#12 0x000055a8be3d8bdf in virtio_blk_resize (opaque=0x55a8c03fa2f0) at /home/kwolf/source/qemu/hw/block/virtio-blk.c:1089
#13 0x000055a8be8424b8 in blk_root_resize (child=0x55a8c033cb10) at block/block-backend.c:1043
#14 0x000055a8be85e1b5 in bdrv_parent_cb_resize (bs=0x55a8c03806b0) at block/io.c:3053
#15 0x000055a8be85a5e0 in bdrv_co_write_req_finish (child=0x55a8c0354120, offset=10737418240, bytes=10737418240, req=0x7f6deddcdef0, ret=0) at block/io.c:1604
#16 0x000055a8be85e5e6 in bdrv_co_truncate (child=0x55a8c0354120, offset=21474836480, prealloc=PREALLOC_MODE_OFF, errp=0x7fffbcda5f40) at block/io.c:3139
#17 0x000055a8be85e65a in bdrv_truncate_co_entry (opaque=0x7fffbcda5e30) at block/io.c:3159
#18 0x000055a8be9472fc in coroutine_trampoline (i0=-1078091680, i1=21928) at util/coroutine-ucontext.c:116
#19 0x00007f6df9820250 in __start_context () at /lib64/libc.so.6
#20 0x00007fffbcda5460 in  ()
#21 0x0000000000000000 in  ()

Thread 1 (Thread 0x7f6deef41680 (LWP 12618)):
#0  0x00007f6df99a9b8d in __lll_lock_wait () at /lib64/libpthread.so.0
#1  0x00007f6df99a2e74 in pthread_mutex_lock () at /lib64/libpthread.so.0
#2  0x000055a8be92b213 in qemu_mutex_lock_impl (mutex=0x55a8bfbcc220, file=0x55a8beb99bc1 "util/async.c", line=511) at util/qemu-thread-posix.c:66
#3  0x000055a8be923946 in aio_context_acquire (ctx=0x55a8bfbcc1c0) at util/async.c:511
#4  0x000055a8be85e7e2 in bdrv_truncate (child=0x55a8c0354120, offset=21474836480, prealloc=PREALLOC_MODE_OFF, errp=0x7fffbcda5f40) at block/io.c:3182
#5  0x000055a8be844770 in blk_truncate (blk=0x55a8c035eb00, offset=21474836480, prealloc=PREALLOC_MODE_OFF, errp=0x7fffbcda5f40) at block/block-backend.c:2045
#6  0x000055a8be5113e1 in qmp_block_resize (has_device=false, device=0x0, has_node_name=true, node_name=0x55a8c038b7e0 "file_image3", size=21474836480, errp=0x7fffbcda5f40) at blockdev.c:3168
#7  0x000055a8be773596 in qmp_marshal_block_resize (args=0x7f6dd8003c30, ret=0x7fffbcda5fd8, errp=0x7fffbcda5fd0) at qapi/qapi-commands-block-core.c:281
#8  0x000055a8be8cc23e in do_qmp_dispatch (cmds=0x55a8bf123470 <qmp_commands>, request=0x7f6dd8005720, allow_oob=false, errp=0x7fffbcda6078) at qapi/qmp-dispatch.c:131
#9  0x000055a8be8cc43f in qmp_dispatch (cmds=0x55a8bf123470 <qmp_commands>, request=0x7f6dd8005720, allow_oob=false) at qapi/qmp-dispatch.c:174
#10 0x000055a8be377ff3 in monitor_qmp_dispatch (mon=0x55a8bfaf8e90, req=0x7f6dd8005720) at /home/kwolf/source/qemu/monitor.c:4141
#11 0x000055a8be37837c in monitor_qmp_bh_dispatcher (data=0x0) at /home/kwolf/source/qemu/monitor.c:4224
#12 0x000055a8be922bed in aio_bh_call (bh=0x55a8bfa78da0) at util/async.c:90
#13 0x000055a8be922c85 in aio_bh_poll (ctx=0x55a8bfa78c60) at util/async.c:118
#14 0x000055a8be927acc in aio_dispatch (ctx=0x55a8bfa78c60) at util/aio-posix.c:460
#15 0x000055a8be923020 in aio_ctx_dispatch (source=0x55a8bfa78c60, callback=0x0, user_data=0x0) at util/async.c:261
#16 0x00007f6dfbd5e06d in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#17 0x000055a8be9263fd in glib_pollfds_poll () at util/main-loop.c:213
#18 0x000055a8be926477 in os_host_main_loop_wait (timeout=29876310) at util/main-loop.c:236
#19 0x000055a8be92657c in main_loop_wait (nonblocking=0) at util/main-loop.c:512
#20 0x000055a8be51ef96 in main_loop () at vl.c:1868
#21 0x000055a8be526399 in main (argc=27, argv=0x7fffbcda6678, envp=0x7fffbcda6758) at vl.c:4510

--- Additional comment from Xueqiang Wei on 2019-06-05 08:51:10 UTC ---

with same steps via virtio-scsi disk, not hit this issue.


{"execute": "blockdev-add","arguments": {"node-name":"data_disk","driver":"file","filename":"/home/nfs_test/data.qcow2"}}
{"execute": "blockdev-add","arguments": {"node-name": "disk1","driver":"qcow2","file":"data_disk"}}
{"execute":"device_add","arguments":{"driver":"virtio-scsi-pci","id":"virtio_scsi_pci1","bus":"pcie.0-root-port-9","addr":"0x0","iothread":"iothread0"}}
{"execute":"device_add","arguments":{"driver":"scsi-hd","drive":"disk1","id":"data_disk","bus":"virtio_scsi_pci1.0"}}



{"execute":"block_resize","arguments":{"node-name":"disk1","size":21474836480 }}

{"return": {}}

--- Additional comment from Xueqiang Wei on 2019-06-05 10:00:20 UTC ---

(In reply to Xueqiang Wei from comment #4)
> with same steps via virtio-scsi disk, not hit this issue.
> 
> 
> {"execute": "blockdev-add","arguments":
> {"node-name":"data_disk","driver":"file","filename":"/home/nfs_test/data.
> qcow2"}}
> {"execute": "blockdev-add","arguments": {"node-name":
> "disk1","driver":"qcow2","file":"data_disk"}}
> {"execute":"device_add","arguments":{"driver":"virtio-scsi-pci","id":
> "virtio_scsi_pci1","bus":"pcie.0-root-port-9","addr":"0x0","iothread":
> "iothread0"}}
> {"execute":"device_add","arguments":{"driver":"scsi-hd","drive":"disk1","id":
> "data_disk","bus":"virtio_scsi_pci1.0"}}
> 
> 
> 
> {"execute":"block_resize","arguments":{"node-name":"disk1","size":
> 21474836480 }}
> 
> {"return": {}}


Retested with 16T disk, hit this issue.


Details:

Host:
kernel-4.18.0-85.el8.x86_64
qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71

Guest:
win2019 with virtio-win-prewhql-0.1-171

1. Create 16T image on host.
# qemu-img create -f qcow2 /home/nfs_test/data.qcow2 16T

2. boot guest with iothread
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_w2u90exl/monitor-qmpmonitor1-20181127-024837-wdAVx2FL,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_w2u90exl/monitor-catch_monitor-20181127-024837-wdAVx2FL,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idulvcka  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_w2u90exl/serial-serial0-20181127-024837-wdAVx2FL,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181127-024837-wdAVx2FL,path=/var/tmp/avocado_w2u90exl/seabios-20181127-024837-wdAVx2FL,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181127-024837-wdAVx2FL,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
    -object iothread,id=iothread0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \
    -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/nfs_test/win2019-64-virtio-scsi.qcow2,node-name=my_file \
    -blockdev driver=qcow2,node-name=my,file=my_file,cache.direct=on,cache.no-flush=off \
    -device scsi-hd,drive=my,bus=virtio_scsi_pci0.0,write-cache=on \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:34:35:36:37:38,id=idyb3F88,vectors=4,netdev=idTAFS0s,bus=pcie.0-root-port-7,addr=0x0  \
    -netdev tap,id=idTAFS0s,vhost=on \
    -m 8G  \
    -smp 12,maxcpus=12,cores=6,threads=1,sockets=2  \
    -cpu 'Opteron_G5',hv_stimer,hv_synic,hv_vpindex,hv_reset,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv-tlbflush,+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie.0-root-port-9,slot=9,chassis=9,addr=0x9,bus=pcie.0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:4444,server,nowait \

3. Hotplug the disk via qmp:

# telnet localhost 4444

{"execute":"qmp_capabilities"}
{"execute": "query-block"}

{"execute": "blockdev-add","arguments": {"node-name":"data_disk","driver":"file","filename":"/home/nfs_test/data.qcow2"}}
{"execute": "blockdev-add","arguments": {"node-name": "disk1","driver":"qcow2","file":"data_disk"}}
{"execute":"device_add","arguments":{"driver":"virtio-scsi-pci","id":"virtio_scsi_pci1","bus":"pcie.0-root-port-9","addr":"0x0","iothread":"iothread0"}}
{"execute":"device_add","arguments":{"driver":"scsi-hd","drive":"disk1","id":"data_disk","bus":"virtio_scsi_pci1.0"}}

{"execute": "query-block"}

4. block_resize (shrink to 3G)

{"execute":"block_resize","arguments":{"node-name":"disk1","size":3073741824 }}


after step 4, guest hang.

# pstack 28508
Thread 21 (Thread 0x7f9983fff700 (LWP 28751)):
#0  0x00007f9be86c9082 in do_futex_wait () at /lib64/libpthread.so.0
#1  0x00007f9be86c9193 in __new_sem_wait_slow () at /lib64/libpthread.so.0
#2  0x00005613d95538df in qemu_sem_timedwait (sem=sem@entry=0x7f9bd4003ab8, ms=ms@entry=10000) at util/qemu-thread-posix.c:289
#3  0x00005613d954e764 in worker_thread (opaque=0x7f9bd4003a40) at util/thread-pool.c:92
#4  0x00005613d9553314 in qemu_thread_start (args=0x7f9958000b20) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 20 (Thread 0x7f9990adf700 (LWP 28750)):
#0  0x00007f9be86c9082 in do_futex_wait () at /lib64/libpthread.so.0
#1  0x00007f9be86c9193 in __new_sem_wait_slow () at /lib64/libpthread.so.0
#2  0x00005613d95538df in qemu_sem_timedwait (sem=sem@entry=0x7f9bd4003ab8, ms=ms@entry=10000) at util/qemu-thread-posix.c:289
#3  0x00005613d954e764 in worker_thread (opaque=0x7f9bd4003a40) at util/thread-pool.c:92
#4  0x00005613d9553314 in qemu_thread_start (args=0x7f996c000b20) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 19 (Thread 0x7f99931ff700 (LWP 28749)):
#0  0x00007f9be86c9082 in do_futex_wait () at /lib64/libpthread.so.0
#1  0x00007f9be86c9193 in __new_sem_wait_slow () at /lib64/libpthread.so.0
#2  0x00005613d95538df in qemu_sem_timedwait (sem=sem@entry=0x7f9bd4003ab8, ms=ms@entry=10000) at util/qemu-thread-posix.c:289
#3  0x00005613d954e764 in worker_thread (opaque=0x7f9bd4003a40) at util/thread-pool.c:92
#4  0x00005613d9553314 in qemu_thread_start (args=0x7f9948000b20) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 18 (Thread 0x7f99827fc700 (LWP 28748)):
#0  0x00007f9be83e3f7f in fdatasync () at /lib64/libc.so.6
#1  0x00005613d954c409 in qemu_fdatasync (fd=<optimized out>) at util/cutils.c:159
#2  0x00005613d94c92da in handle_aiocb_flush (opaque=<optimized out>) at block/file-posix.c:1223
#3  0x00005613d954e7ec in worker_thread (opaque=0x7f9bd4003a40) at util/thread-pool.c:105
#4  0x00005613d9553314 in qemu_thread_start (args=0x7f9bd401ec20) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 17 (Thread 0x7f9993dff700 (LWP 28534)):
#0  0x00007f9be86c64dc in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00005613d95536cd in qemu_cond_wait_impl (cond=<optimized out>, mutex=0x5613db9c8798, file=0x5613d96cb897 "ui/vnc-jobs.c", line=214) at util/qemu-thread-posix.c:161
#2  0x00005613d947eed1 in vnc_worker_thread_loop (queue=queue@entry=0x5613db9c8760) at ui/vnc-jobs.c:214
#3  0x00005613d947f490 in vnc_worker_thread (arg=0x5613db9c8760) at ui/vnc-jobs.c:324
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da9a9ca0) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 16 (Thread 0x7f9ba2dff700 (LWP 28532)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da438f90, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da438f90) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da45b640) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 15 (Thread 0x7f9ba37fe700 (LWP 28531)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da4160a0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da4160a0) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da438750) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 14 (Thread 0x7f9ba3fff700 (LWP 28530)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da3f31b0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da3f31b0) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da415860) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 13 (Thread 0x7f9bb93ff700 (LWP 28529)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da3d02c0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da3d02c0) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da3f2970) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 12 (Thread 0x7f9bb9fff700 (LWP 28528)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da3ad3d0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da3ad3d0) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da3cfa80) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 11 (Thread 0x7f9bbabff700 (LWP 28527)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da389670, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da389670) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da3acb90) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 10 (Thread 0x7f9bbb7fe700 (LWP 28526)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da366780, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da366780) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da388e30) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 9 (Thread 0x7f9bbbfff700 (LWP 28525)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da343890, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da343890) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da365f40) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 8 (Thread 0x7f9bd11ff700 (LWP 28524)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da3207c0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da3207c0) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da343080) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 7 (Thread 0x7f9bd1dff700 (LWP 28523)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da2fd4b0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da2fd4b0) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da31ff80) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 6 (Thread 0x7f9bd29ff700 (LWP 28522)):
#0  0x00007f9be83e304b in ioctl () at /lib64/libc.so.6
#1  0x00005613d926a2c9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5613da2d9ac0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2159
#2  0x00005613d926a389 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1996
#3  0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da2d9ac0) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da2fcc70) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 5 (Thread 0x7f9bd343c700 (LWP 28521)):
#0  0x00007f9be86c98ed in __lll_lock_wait () at /lib64/libpthread.so.0
#1  0x00007f9be86c2b59 in pthread_mutex_lock () at /lib64/libpthread.so.0
#2  0x00005613d95533fd in qemu_mutex_lock_impl (mutex=0x5613d9da2e60 <qemu_global_mutex>, file=0x5613d95ec9b0 "/builddir/build/BUILD/qemu-4.0.0/exec.c", line=3243) at util/qemu-thread-posix.c:66
#3  0x00005613d92423ee in qemu_mutex_lock_iothread_impl (file=<optimized out>, line=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1850
#4  0x00005613d91fdc09 in prepare_mmio_access (mr=<optimized out>, mr=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/exec.c:3243
#5  0x00005613d9202b7f in flatview_read_continue (fv=0x7f9bc40d9310, addr=1544, attrs=..., buf=<optimized out>, len=4, addr1=<optimized out>, l=<optimized out>, mr=0x5613da9a4470) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/exec.c:3338
#6  0x00005613d9202da3 in flatview_read (fv=0x7f9bc40d9310, addr=1544, attrs=..., buf=0x7f9bedbc1000 "", len=4) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/exec.c:3378
#7  0x00005613d9202ecf in address_space_read_full (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/exec.c:3391
#8  0x00005613d926a5c4 in kvm_handle_io (count=1, size=4, direction=<optimized out>, data=<optimized out>, attrs=..., port=1544) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:1790
#9  0x00005613d926a5c4 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/accel/kvm/kvm-all.c:2036
#10 0x00005613d92425b6 in qemu_kvm_cpu_thread_fn (arg=0x5613da28acb0) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/cpus.c:1281
#11 0x00005613d9553314 in qemu_thread_start (args=0x5613da2aec70) at util/qemu-thread-posix.c:502
#12 0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#13 0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 4 (Thread 0x7f9bd3c3d700 (LWP 28520)):
#0  0x00007f9be83e13d1 in poll () at /lib64/libc.so.6
#1  0x00007f9bed26d926 in g_main_context_iterate.isra () at /lib64/libglib-2.0.so.0
#2  0x00007f9bed26dce2 in g_main_loop_run () at /lib64/libglib-2.0.so.0
#3  0x00005613d933fbd1 in iothread_run (opaque=0x5613da22b760) at iothread.c:82
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da26e450) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 3 (Thread 0x7f9be10d6700 (LWP 28510)):
#0  0x00007f9be83e14c6 in ppoll () at /lib64/libc.so.6
#1  0x00005613d954f115 in ppoll (__ss=0x0, __timeout=0x7f9be10d5640, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2  0x00005613d954f115 in qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at util/qemu-timer.c:334
#3  0x00005613d9551174 in aio_poll (ctx=0x5613da246ff0, blocking=blocking@entry=true) at util/aio-posix.c:666
#4  0x00005613d933fba4 in iothread_run (opaque=0x5613da22f400) at iothread.c:75
#5  0x00005613d9553314 in qemu_thread_start (args=0x5613da247d30) at util/qemu-thread-posix.c:502
#6  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#7  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 2 (Thread 0x7f9be18d7700 (LWP 28509)):
#0  0x00007f9be83e6ead in syscall () at /lib64/libc.so.6
#1  0x00005613d9553b3f in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at util/qemu-thread-posix.c:438
#2  0x00005613d9553b3f in qemu_event_wait (ev=ev@entry=0x5613d9dd7d48 <rcu_call_ready_event>) at util/qemu-thread-posix.c:442
#3  0x00005613d95650d2 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:261
#4  0x00005613d9553314 in qemu_thread_start (args=0x5613da1711e0) at util/qemu-thread-posix.c:502
#5  0x00007f9be86c02de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f9be83ec653 in clone () at /lib64/libc.so.6
Thread 1 (Thread 0x7f9bedb8fec0 (LWP 28508)):
#0  0x00007f9be83e14c6 in ppoll () at /lib64/libc.so.6
#1  0x00005613d954f159 in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2  0x00005613d954f159 in qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at util/qemu-timer.c:322
#3  0x00005613d9551174 in aio_poll (ctx=0x5613da236a50, blocking=blocking@entry=true) at util/aio-posix.c:666
#4  0x00005613d94d56ea in bdrv_truncate (child=<optimized out>, offset=offset@entry=3073741824, prealloc=prealloc@entry=PREALLOC_MODE_OFF, errp=errp@entry=0x7ffe380bdb98) at block/io.c:3203
#5  0x00005613d94c1daf in blk_truncate (blk=blk@entry=0x5613db36fa40, offset=offset@entry=3073741824, prealloc=prealloc@entry=PREALLOC_MODE_OFF, errp=errp@entry=0x7ffe380bdb98) at block/block-backend.c:1978
#6  0x00005613d933c4d6 in qmp_block_resize (has_device=<optimized out>, device=0x0, has_node_name=<optimized out>, node_name=<optimized out>, size=3073741824, errp=errp@entry=0x7ffe380bdb98) at blockdev.c:3147
#7  0x00005613d9449dc7 in qmp_marshal_block_resize (args=<optimized out>, ret=<optimized out>, errp=0x7ffe380bdc18) at qapi/qapi-commands-block-core.c:281
#8  0x00005613d95073dc in do_qmp_dispatch (errp=0x7ffe380bdc10, allow_oob=<optimized out>, request=<optimized out>, cmds=0x5613d9da2f30 <qmp_commands>) at qapi/qmp-dispatch.c:131
#9  0x00005613d95073dc in qmp_dispatch (cmds=0x5613d9da2f30 <qmp_commands>, request=<optimized out>, allow_oob=<optimized out>) at qapi/qmp-dispatch.c:174
#10 0x00005613d9246282 in monitor_qmp_dispatch (mon=0x5613da289e50, req=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/monitor.c:4124
#11 0x00005613d924c209 in monitor_qmp_bh_dispatcher (data=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64/monitor.c:4207
#12 0x00005613d954dbc6 in aio_bh_call (bh=0x5613da1bfc60) at util/async.c:118
#13 0x00005613d954dbc6 in aio_bh_poll (ctx=ctx@entry=0x5613da1be840) at util/async.c:118
#14 0x00005613d9551004 in aio_dispatch (ctx=0x5613da1be840) at util/aio-posix.c:460
#15 0x00005613d954daa2 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261
#16 0x00007f9bed26d5ed in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#17 0x00005613d95501e8 in glib_pollfds_poll () at util/main-loop.c:213
#18 0x00005613d95501e8 in os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:236
#19 0x00005613d95501e8 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:512
#20 0x00005613d9345579 in main_loop () at vl.c:1988
#21 0x00005613d91fa078 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4637