Bug 2058457

Summary: Qemu core dump when mirror before "STOP" event received that caused by no space left error (iothread enabled)
Product: Red Hat Enterprise Linux 9 Reporter: aihua liang <aliang>
Component: qemu-kvmAssignee: Hanna Czenczek <hreitz>
qemu-kvm sub component: Block Jobs QA Contact: aihua liang <aliang>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: medium    
Priority: medium CC: coli, hreitz, jinzhao, mdeng, meili, ngu, virt-maint
Version: 9.0Keywords: Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2058459 (view as bug list) Environment:
Last Closed: 2023-06-20 08:20:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2058459    

Description aihua liang 2022-02-25 03:24:51 UTC 
Description of problem:
 Qemu core dump when mirror after block error occured

Version-Release number of selected component (if applicable):


How reproducible:
2/200, random


Steps to Reproduce:
1.Create a small space
  # dd if=/dev/urandom of=/tmp/tmp_image_file bs=1M count=100
  # mkfs.ext4 /tmp/tmp_image_file
  # mkdir -p /tmp/tmp_target_path
  # mount -o loop /tmp/tmp_image_file /tmp/tmp_target_path

2.Create a 2G data disk on the small space
  #qemu-img create -f qcow2 /tmp/tmp_target_path/data1.qcow2 2G

3.Start guest with data disk.
  /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
    -cpu 'Cascadelake-Server',ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,kvm_pv_unhalt=on \
    -chardev socket,path=/tmp/avocado_5f_ewkop/monitor-qmpmonitor1-20220224-210926-WSlttjSU,id=qmp_id_qmpmonitor1,wait=off,server=on  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/tmp/avocado_5f_ewkop/monitor-catch_monitor-20220224-210926-WSlttjSU,id=qmp_id_catch_monitor,wait=off,server=on  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idZRqwzD \
    -chardev socket,path=/tmp/avocado_5f_ewkop/serial-serial0-20220224-210926-WSlttjSU,id=chardev_serial0,wait=off,server=on \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20220224-210926-WSlttjSU,path=/tmp/avocado_5f_ewkop/seabios-20220224-210926-WSlttjSU,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20220224-210926-WSlttjSU,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel860-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -blockdev node-name=file_data1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/tmp/tmp_target_path/data1.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_data1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_data1 \
    -device scsi-hd,id=data1,drive=drive_data1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:37:23:84:f0:e3,id=id1PEaqm,netdev=idCeYpPm,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idCeYpPm,vhost=on,vhostfd=23,fd=7  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5
    
4.Create target node.
   {"execute": "blockdev-create", "arguments": {"options": {"driver": "file", "filename": "/root/avocado/data/avocado-vt/mirror1.qcow2", "size": 2147483648}, "job-id": "file_mirror1"}, "id": "yDszJDGW"}
   {"execute": "job-dismiss", "arguments": {"id": "file_mirror1"}, "id": "hw1F2DM8"}
   {"execute": "blockdev-add", "arguments": {"node-name": "file_mirror1", "driver": "file", "filename": "/root/avocado/data/avocado-vt/mirror1.qcow2", "aio": "threads", "auto-read-only": true, "discard": "unmap"}, "id": "IhEOPdrY"}
   {"execute": "blockdev-create", "arguments": {"options": {"driver": "qcow2", "file": "file_mirror1", "size": 2147483648}, "job-id": "drive_mirror1"}, "id": "DFNuXOQJ"}
   {"execute": "job-dismiss", "arguments": {"id": "drive_mirror1"}, "id": "Oekl91Yc"}
   {"execute": "blockdev-add", "arguments": {"node-name": "drive_mirror1", "driver": "qcow2", "file": "file_mirror1", "read-only": false}, "id": "8HdIDz0D"}

5.Write 200M file on data disk
  (guest)#dd if=/dev/urandom of=/dev/sdb/file bs=1M count=200
  {"timestamp": {"seconds": 1645701944, "microseconds": 422050}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}
  {"timestamp": {"seconds": 1645701944, "microseconds": 422113}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}
  {"timestamp": {"seconds": 1645701944, "microseconds": 422143}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}

6.Mirror from src to dst before "STOP" event received in qmp.
  {"execute": "blockdev-mirror", "arguments": {"sync": "full", "device": "drive_data1", "target": "drive_mirror1", "job-id": "drive_data1_rjew"}, "id": "wjTQ0RSx"}
  {"timestamp": {"seconds": 1645701944, "microseconds": 452273}, "event": "STOP"}
  {"timestamp": {"seconds": 1645701944, "microseconds": 456855}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "drive_data1_ccja"}}
  {"timestamp": {"seconds": 1645701944, "microseconds": 456916}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "drive_data1_ccja"}}
  {"return": {}, "id": "YK6FvYqe"}
  {"timestamp": {"seconds": 1645701944, "microseconds": 517644}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "drive_data1_ccja"}}
  {"timestamp": {"seconds": 1645701944, "microseconds": 517722}, "event": "BLOCK_JOB_READY", "data": {"device": "drive_data1_ccja", "len": 2147483648, "offset": 2147483648, "speed": 0, "type": "mirror"}}
  
7. When mirror reach ready status, complte the mirror job.
  {"execute": "job-complete", "arguments": {"id": "drive_data1_ccja"}, "id": "Qgn0YYDv"}
  {"return": {}, "id": "Qgn0YYDv"}
  {"execute": "query-jobs", "id": "GRnhZgX3"}
  {"timestamp": {"seconds": 1645701944, "microseconds": 560376}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "drive_data1_ccja"}}
  {"timestamp": {"seconds": 1645701944, "microseconds": 560405}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "drive_data1_ccja"}}

Actual results:
 After step7, qemu core dump.
 gdb info as bellow:
   Stack trace of thread 327974:
                #0  0x000055f126111bba bdrv_can_set_aio_context (qemu-kvm + 0x799bba)
                #1  0x000055f1261134ae bdrv_attach_child_common (qemu-kvm + 0x79b4ae)
                #2  0x000055f126113212 bdrv_root_attach_child (qemu-kvm + 0x79b212)
                #3  0x000055f12613c33d blk_insert_bs (qemu-kvm + 0x7c433d)
                #4  0x000055f12615aa0a mirror_exit_common (qemu-kvm + 0x7e2a0a)
                #5  0x000055f126122a87 job_prepare (qemu-kvm + 0x7aaa87)
                #6  0x000055f126123980 job_completed (qemu-kvm + 0x7ab980)
                #7  0x000055f1261235b9 job_exit (qemu-kvm + 0x7ab5b9)
                #8  0x000055f1262dea81 aio_dispatch (qemu-kvm + 0x966a81)
                #9  0x000055f1262f8cb2 aio_ctx_dispatch (qemu-kvm + 0x980cb2)
                #10 0x00007ff18dbd0d4f g_main_context_dispatch (libglib-2.0.so.0 + 0x54d4f)
                #11 0x000055f126305973 main_loop_wait (qemu-kvm + 0x98d973)
                #12 0x000055f125f7f127 qemu_main_loop (qemu-kvm + 0x607127)
                #13 0x000055f125cd56f2 main (qemu-kvm + 0x35d6f2)
                #14 0x00007ff18d8b3e50 __libc_start_call_main (libc.so.6 + 0x44e50)
                #15 0x00007ff18d8b3efc __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x44efc)
                #16 0x000055f125cd4db5 _start (qemu-kvm + 0x35cdb5)
                
                Stack trace of thread 327976:
                #0  0x00007ff18d9b6fde ppoll (libc.so.6 + 0x147fde)
                #1  0x000055f1262dffe2 fdmon_poll_wait (qemu-kvm + 0x967fe2)
                #2  0x000055f1262df774 aio_poll (qemu-kvm + 0x967774)
                #3  0x000055f12610a096 iothread_run (qemu-kvm + 0x792096)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327975:
                #0  0x00007ff18d910450 __GI___lll_lock_wait (libc.so.6 + 0xa1450)
                #1  0x00007ff18d916b12 __pthread_mutex_lock.5 (libc.so.6 + 0xa7b12)
                #2  0x000055f1262e29af qemu_mutex_lock_impl (qemu-kvm + 0x96a9af)
                #3  0x000055f1262ef681 call_rcu_thread (qemu-kvm + 0x977681)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327981:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327980:
                #0  0x00007ff18d9b6edf __poll (libc.so.6 + 0x147edf)
                #1  0x00007ff18dc2559c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa959c)
                #2  0x00007ff18dbd0463 g_main_loop_run (libglib-2.0.so.0 + 0x54463)
                #3  0x000055f12610a0b4 iothread_run (qemu-kvm + 0x7920b4)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327983:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327982:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327986:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327984:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327988:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327987:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327985:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327989:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328076:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327990:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125f6e567 qemu_wait_io_event (qemu-kvm + 0x5f6567)
                #4  0x000055f12608b411 kvm_vcpu_thread_fn (qemu-kvm + 0x713411)
                #5  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #6  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #7  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328053:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 327992:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d912a60 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa3a60)
                #2  0x000055f1262e2e8f qemu_cond_wait_impl (qemu-kvm + 0x96ae8f)
                #3  0x000055f125d12657 vnc_worker_thread (qemu-kvm + 0x39a657)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328077:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328086:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328096:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328100:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328085:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328080:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328101:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328087:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328091:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328079:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328074:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328097:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328089:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328088:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328090:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328084:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328092:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328082:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328095:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328073:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328083:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328052:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328093:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328081:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328094:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328099:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328075:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328098:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                
                Stack trace of thread 328078:
                #0  0x00007ff18d9102ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                #1  0x00007ff18d91b443 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac443)
                #2  0x000055f1262e329b qemu_sem_timedwait (qemu-kvm + 0x96b29b)
                #3  0x000055f126309e46 worker_thread (qemu-kvm + 0x991e46)
                #4  0x000055f1262e382a qemu_thread_start (qemu-kvm + 0x96b82a)
                #5  0x00007ff18d9136ca start_thread (libc.so.6 + 0xa46ca)
                #6  0x00007ff18d8b33f0 __clone3 (libc.so.6 + 0x443f0)
                ELF object binary architecture: AMD x86-64

Note:
 When do mirror after "STOP" event received, can't hit the core dump issue and mirror works ok.

Expected results:
 Mirror can executed successfully.

Additional info:
Comment 3 aihua liang 2022-02-25 08:16:03 UTC 
qemu-kvm-6.2.0-1.el9 also hit this issue after 50 times automation test.
Comment 4 Hanna Czenczek 2022-02-25 12:57:50 UTC 
I haven’t yet been able to reproduce this, I’ll try further next weeek.

Given that this seems to happen only very rarely, and only under very specific circumstances, I wouldn’t at this point say this has very high priority.  Furthermore, the stack trace points into AioContext management, which from experience is always hairy to fix, so even if I found what the problem was, a fix might take more time.  Considering where we are in the release cycle, I believe we should put this off for 9.1.
Comment 6 aihua liang 2022-03-11 10:34:02 UTC 
One key point to reproduce this issue is: iothread enable.

Test on qemu-kvm-6.2.0-11.el9, hit this issue with reproduce ratio:3/20.
From qmp logs, we can see that cases always failed when blockdev-mirror executed before "event": "STOP", details as bellow:

QMP log of all passed cases:
{"timestamp": {"seconds": 1646991300, "microseconds": 451869}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 451936}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 451971}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 484348}, "event": "STOP"}
  {"execute": "blockdev-mirror", "arguments": {"sync": "full", "device": "drive_data1", "target": "drive_mirror1", "job-id": "drive_data1_WK4j"}, "id": "JpBF8Xab"}
{"timestamp": {"seconds": 1646991300, "microseconds": 642861}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "drive_data1_WK4j"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 642918}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "drive_data1_WK4j"}}
{"return": {}, "id": "JpBF8Xab"}
{"timestamp": {"seconds": 1646991300, "microseconds": 702924}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "drive_data1_WK4j"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 702982}, "event": "BLOCK_JOB_READY", "data": {"device": "drive_data1_WK4j", "len": 2147483648, "offset": 2147483648, "speed": 0, "type": "mirror"}}
  {"execute": "job-complete", "arguments": {"id": "drive_data1_WK4j"}, "id": "78imD7Fa"}
{"return": {}, "id": "78imD7Fa"}
{"timestamp": {"seconds": 1646991300, "microseconds": 746434}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "drive_data1_WK4j"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 746464}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "drive_data1_WK4j"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 746537}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive_data1_WK4j", "len": 2147483648, "offset": 2147483648, "speed": 0, "type": "mirror"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 746564}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "drive_data1_WK4j"}}
{"timestamp": {"seconds": 1646991300, "microseconds": 746583}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "drive_data1_WK4j"}}

QMP log of all failed 3 cases:
{"timestamp": {"seconds": 1646991988, "microseconds": 496256}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}
{"timestamp": {"seconds": 1646991988, "microseconds": 496324}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}
{"timestamp": {"seconds": 1646991988, "microseconds": 496354}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": true, "node-name": "drive_data1", "reason": "No space left on device", "operation": "write", "action": "stop"}}
  {"execute": "blockdev-mirror", "arguments": {"sync": "full", "device": "drive_data1", "target": "drive_mirror1", "job-id": "drive_data1_bvUb"}, "id": "hO1GGkhv"}
{"timestamp": {"seconds": 1646991988, "microseconds": 530324}, "event": "STOP"}
{"timestamp": {"seconds": 1646991988, "microseconds": 530884}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "drive_data1_bvUb"}}
{"timestamp": {"seconds": 1646991988, "microseconds": 530937}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "drive_data1_bvUb"}}
{"return": {}, "id": "hO1GGkhv"}
{"timestamp": {"seconds": 1646991988, "microseconds": 591438}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "drive_data1_bvUb"}}
{"timestamp": {"seconds": 1646991988, "microseconds": 591490}, "event": "BLOCK_JOB_READY", "data": {"device": "drive_data1_bvUb", "len": 2147483648, "offset": 2147483648, "speed": 0, "type": "mirror"}}
  {"execute": "job-complete", "arguments": {"id": "drive_data1_bvUb"}, "id": "tdxHfkzJ"}
{"return": {}, "id": "tdxHfkzJ"}
{"timestamp": {"seconds": 1646991988, "microseconds": 634628}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "drive_data1_bvUb"}}
{"timestamp": {"seconds": 1646991988, "microseconds": 634656}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "drive_data1_bvUb"}}

So I think if we can make "event": "STOP" send out a little late, just after blockdev-mirror, we can reproduce this issue easily.
Comment 7 Hanna Czenczek 2022-03-11 11:09:45 UTC 
Oh, sorry, I didn’t write that I’ve been able to reproduce this in the meantime.

The problem appears to be as such:
- The I/O error causes a VM STOP to be requested
- main_loop_should_exit() sees the request, clears it, and invokes vm_stop()
- vm_stop() runs, nothing out of the ordinary happens
- In the meantime, before the VM is actually stopped, the guest block device will see more I/O errors and request STOP again (I suspect this will only happen with iothread)
- During the next iteration of main_loop_should_exit(), the monitor makes internal progress and gets ready to process the next incoming QMP request (the blockdev-mirror command), but doesn’t yet
- main_loop_should_exit() sees the new STOP request, and invokes vm_stop() again
- vm_stop() calls do_vm_stop(), which will flush all block devices
- bdrv_flush_all() uses bdrv_next() to iterate over all root BDSs, flushing them
- bdrv_next() goes over each guest device’s BlockBackend, taking the root BDS inserted into it, bdrv_ref()-ing it (so it stays valid throughout the iteration), and returns it
- When we come to the block device in the I/O thread and flush it, the main thread is unlocked and polled (bdrv_poll_co() in bdrv_flush()), and so the monitor can begin to process the incoming blockdev-mirror QMP command
- When flushing is complete, and bdrv_co_flush_entry() returns, the QMP processing BH will be run first (instead of returning to bdrv_flush()), and so now the mirror job starts up, changing the block graph, specifically replacing the root BDS inserted into the BlockBackend in the I/O thread (the one which bdrv_flush_all() is currently trying to flush)
- bdrv_flush_all() continues, going to bdrv_next() to flush the next BDS
- bdrv_next() fetches the BB it visited last, bdrv_unref()s the root BDS inserted into it, assuming that’s the BDS it returned last time; but it’s a different BDS now that the mirror job has changed it, so we that new BDS’s reference count will be one too low from now on (this BDS is the mirror job’s filter node)
- When the mirror job’s filter node should reach a refcount of 1 (when the mirror job completes), it’s actually freed, so any subsequent access to it will be a use-after-free and can cause qemu to crash (and one of them certainly will)

We aren’t sure how to solve the problem yet.  I’ve sent a patch upstream to make bdrv_next() resilient to such graph changes, but Kevin doesn’t like it much; he’d prefer to put the bdrv_flush_all() into a drained section, and make drained section pause QMP processing (but that’s complicated).

We could also prevent the second vm_stop() from flushing (because the VM is already stopped by this point so we shouldn’t need to flush), but we’ll just need to have this discussion upstream.
Comment 9 aihua liang 2022-05-16 09:52:45 UTC 
Test with qemu-*.el9, all hit this issue.
qemu-kvm-7.0.0-1.el9: 9/300(run 300times, hit coredump 9 times)
qemu-kvm-6.2.0-1.el9: 5/200(run 200times, hit coredump 5 times)
qemu-kvm-6.1.0-1.el9: 1/8 (catch the coredump on the 8th test)
qemu-kvm-5.2.0-2.el9:1/21 (catch the coredump on the 21st test)
qemu-kvm-5.2.0-rc1.1.el9:1/6(catch the coredump on the 6th test)

Will do more on RHEL8 to check if it's a regression.
Comment 10 aihua liang 2022-05-18 07:16:45 UTC 
Sorry, it's not a regression issue.
Check it on RHEL8.6, RHEL8.5-av, RHEL8.4-av, all hit the coredump issue.
   qemu-kvm-6.2.0-11.module+el8.6.0+14707+5aa4b42d： 1/20 (the 20th run hit the coredump issue)
   qemu-kvm-6.2.0-1.module+el8.6.0+13725+61ae1949： 1/89  (the 89th run hit the coredump issue)
   qemu-kvm-6.0.0-33.module+el8.5.0+13514+2c386966.1: 1/29 (the 29th run hit the coredump issue)
   qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d: 1/45 (the 45th run hit the coredump issue)
   qemu-kvm-5.2.0-16.module+el8.4.0+11721+c8bbc1be.3: 1/28 (the 28th run hit the coredump issue)
   qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9: 1/58 (the 58th run hit the coredump issue)
Comment 11 Klaus Heinrich Kiwi 2022-05-25 11:29:59 UTC 
(In reply to aihua liang from comment #10)
> Sorry, it's not a regression issue.
> Check it on RHEL8.6, RHEL8.5-av, RHEL8.4-av, all hit the coredump issue.
>    qemu-kvm-6.2.0-11.module+el8.6.0+14707+5aa4b42d： 1/20 (the 20th run hit
> the coredump issue)
>    qemu-kvm-6.2.0-1.module+el8.6.0+13725+61ae1949： 1/89  (the 89th run hit
> the coredump issue)
>    qemu-kvm-6.0.0-33.module+el8.5.0+13514+2c386966.1: 1/29 (the 29th run hit
> the coredump issue)
>    qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d: 1/45 (the 45th run hit
> the coredump issue)
>    qemu-kvm-5.2.0-16.module+el8.4.0+11721+c8bbc1be.3: 1/28 (the 28th run hit
> the coredump issue)
>    qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9: 1/58 (the 58th run hit the
> coredump issue)

Let's lower the severity and strip the ITR for the time being. If we happen to have a fix for 9.1, that's better-than-planned-for.
Comment 12 aihua liang 2023-02-03 02:21:25 UTC 
Test on qemu-kvm-7.2.0-5.el9, still hit this issue.

New coredump info:
   Stack trace of thread 83892:
                #0  0x00007f7880ea154c __pthread_kill_implementation (libc.so.6 + 0xa154c)
                #1  0x00007f7880e54d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007f7880e287f3 abort (libc.so.6 + 0x287f3)
                #3  0x00007f7880e29130 __libc_message.cold (libc.so.6 + 0x29130)
                #4  0x00007f7880eab617 malloc_printerr (libc.so.6 + 0xab617)
                #5  0x00007f7880eac32c malloc_consolidate (libc.so.6 + 0xac32c)
                #6  0x00007f7880eadf18 _int_malloc (libc.so.6 + 0xadf18)
                #7  0x00007f7880eb00fe __libc_calloc (libc.so.6 + 0xb00fe)
                #8  0x00007f7881320b41 g_malloc0 (libglib-2.0.so.0 + 0x5ab41)
                #9  0x0000558af0befbc0 qapi_event_send_block_job_completed (qemu-kvm + 0x8dabc0)
                #10 0x0000558af0b031c3 block_job_event_completed_locked (qemu-kvm + 0x7ee1c3)
                #11 0x0000558af0b05d99 job_finalize_single_locked (qemu-kvm + 0x7f0d99)
                #12 0x0000558af0b05a0c job_do_finalize_locked (qemu-kvm + 0x7f0a0c)
                #13 0x0000558af0b06b3b job_exit (qemu-kvm + 0x7f1b3b)
                #14 0x0000558af0ccf4e1 aio_dispatch (qemu-kvm + 0x9ba4e1)
                #15 0x0000558af0cea192 aio_ctx_dispatch (qemu-kvm + 0x9d5192)
                #16 0x00007f788131ae2f g_main_context_dispatch (libglib-2.0.so.0 + 0x54e2f)
                #17 0x0000558af0cebab4 main_loop_wait (qemu-kvm + 0x9d6ab4)
                #18 0x0000558af07f4917 qemu_main_loop (qemu-kvm + 0x4df917)
                #19 0x0000558af067a92a qemu_default_main (qemu-kvm + 0x36592a)
                #20 0x00007f7880e3feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #21 0x00007f7880e3ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #22 0x0000558af067a085 _start (qemu-kvm + 0x365085)
                
                Stack trace of thread 83894:
                #0  0x00007f7880f42abe ppoll (libc.so.6 + 0x142abe)
                #1  0x0000558af0cd09ce fdmon_poll_wait (qemu-kvm + 0x9bb9ce)
                #2  0x0000558af0ccfc0e aio_poll (qemu-kvm + 0x9bac0e)
                #3  0x0000558af0ae8ee2 iothread_run (qemu-kvm + 0x7d3ee2)
                #4  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #5  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83893:
                #0  0x00007f7880f13995 clock_nanosleep.5 (libc.so.6 + 0x113995)
                #1  0x00007f7880f18547 __nanosleep (libc.so.6 + 0x118547)
                #2  0x00007f7881344b07 g_usleep (libglib-2.0.so.0 + 0x7eb07)
                #3  0x0000558af0cdfd0a call_rcu_thread (qemu-kvm + 0x9cad0a)
                #4  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #5  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83899:
                #0  0x00007f7880f429bf __poll (libc.so.6 + 0x1429bf)
                #1  0x00007f788136f49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c)
                #2  0x00007f788131a483 g_main_loop_run (libglib-2.0.so.0 + 0x54483)
                #3  0x0000558af0ae8eff iothread_run (qemu-kvm + 0x7d3eff)
                #4  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #5  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83901:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83900:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83902:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83903:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83904:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83906:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83905:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83907:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83909:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83908:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af07ec60b qemu_wait_io_event (qemu-kvm + 0x4d760b)
                #4  0x0000558af0a96811 kvm_vcpu_thread_fn (qemu-kvm + 0x781811)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83911:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000558af0cd348f qemu_cond_wait_impl (qemu-kvm + 0x9be48f)
                #3  0x0000558af06b0ca6 vnc_worker_thread (qemu-kvm + 0x39bca6)
                #4  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #5  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83978:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83999:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84003:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84001:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84005:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84011:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84008:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84000:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84015:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84026:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84023:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84024:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84012:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84022:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84010:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84021:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83998:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84019:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84025:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84002:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84016:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84014:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84013:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84007:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84006:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84103:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84009:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84020:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84004:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 83979:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84017:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 84018:
                #0  0x00007f7880e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f7880e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x0000558af0cd362c qemu_cond_timedwait_ts (qemu-kvm + 0x9be62c)
                #3  0x0000558af0cd35d0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be5d0)
                #4  0x0000558af0cee397 worker_thread (qemu-kvm + 0x9d9397)
                #5  0x0000558af0cd3eda qemu_thread_start (qemu-kvm + 0x9beeda)
                #6  0x00007f7880e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f7880e3f450 __clone3 (libc.so.6 + 0x3f450)
                ELF object binary architecture: AMD x86-64
Comment 15 aihua liang 2023-03-03 08:02:38 UTC 
Still hit this issue on qemu-kvm-7.2.0-10.el9, coredump info as bellow:
[qemu output] /tmp/aexpect_mSZ6EygM/aexpect-pn0f5jw1.sh: line 1: 128203 Aborted                 (core dumped) MALLOC_PERTURB_=1 /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' ...

Coredump info as bellow:
  Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-5.scope
          Unit: session-5.scope
         Slice: user-0.slice
       Session: 5
     Owner UID: 0 (root)
       Boot ID: bd25cd699f47423e8bcbd376fdbe935c
    Machine ID: 3919555703fd4043b7f3cc2611ad4d18
      Hostname: dell-per740xd-01.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.bd25cd699f47423e8bcbd376fdbe935c.128203.1677829173000000.zst (present)
  Size on Disk: 784.3M
       Message: Process 128203 (qemu-kvm) of user 0 dumped core.
                
                Stack trace of thread 128203:
                #0  0x00007fc902ea154c __pthread_kill_implementation (libc.so.6 + 0xa154c)
                #1  0x00007fc902e54d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007fc902e287f3 abort (libc.so.6 + 0x287f3)
                #3  0x00007fc902e29130 __libc_message.cold (libc.so.6 + 0x29130)
                #4  0x00007fc902eab617 malloc_printerr (libc.so.6 + 0xab617)
                #5  0x00007fc902eac32c malloc_consolidate (libc.so.6 + 0xac32c)
                #6  0x00007fc902eadf18 _int_malloc (libc.so.6 + 0xadf18)
                #7  0x00007fc902eb00fe __libc_calloc (libc.so.6 + 0xb00fe)
                #8  0x00007fc903520b41 g_malloc0 (libglib-2.0.so.0 + 0x5ab41)
                #9  0x000055be060e47d0 qapi_event_send_block_job_completed (qemu-kvm + 0x8de7d0)
                #10 0x000055be05ff4d53 block_job_event_completed_locked (qemu-kvm + 0x7eed53)
                #11 0x000055be05ff7929 job_finalize_single_locked (qemu-kvm + 0x7f1929)
                #12 0x000055be05ff759c job_do_finalize_locked (qemu-kvm + 0x7f159c)
                #13 0x000055be05ff86cb job_exit (qemu-kvm + 0x7f26cb)
                #14 0x000055be061c43b1 aio_dispatch (qemu-kvm + 0x9be3b1)
                #15 0x000055be061df0e2 aio_ctx_dispatch (qemu-kvm + 0x9d90e2)
                #16 0x00007fc90351ae2f g_main_context_dispatch (libglib-2.0.so.0 + 0x54e2f)
                #17 0x000055be061e0a04 main_loop_wait (qemu-kvm + 0x9daa04)
                #18 0x000055be05ce6917 qemu_main_loop (qemu-kvm + 0x4e0917)
                #19 0x000055be05b6c93a qemu_default_main (qemu-kvm + 0x36693a)
                #20 0x00007fc902e3feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #21 0x00007fc902e3ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #22 0x000055be05b6c095 _start (qemu-kvm + 0x366095)
                
                Stack trace of thread 128205:
                #0  0x00007fc902f42abe ppoll (libc.so.6 + 0x142abe)
                #1  0x000055be061c589e fdmon_poll_wait (qemu-kvm + 0x9bf89e)
                #2  0x000055be061c4ade aio_poll (qemu-kvm + 0x9beade)
                #3  0x000055be05fda9c2 iothread_run (qemu-kvm + 0x7d49c2)
                #4  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #5  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128212:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128210:
                #0  0x00007fc902f429bf __poll (libc.so.6 + 0x1429bf)
                #1  0x00007fc90356f49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c)
                #2  0x00007fc90351a483 g_main_loop_run (libglib-2.0.so.0 + 0x54483)
                #3  0x000055be05fda9df iothread_run (qemu-kvm + 0x7d49df)
                #4  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #5  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128218:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128221:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128219:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128220:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128294:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128215:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128214:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128216:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128224:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05ba2ca6 vnc_worker_thread (qemu-kvm + 0x39cca6)
                #4  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #5  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128316:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128211:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128344:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128204:
                #0  0x00007fc902f13995 clock_nanosleep.5 (libc.so.6 + 0x113995)
                #1  0x00007fc902f18547 __nanosleep (libc.so.6 + 0x118547)
                #2  0x00007fc903544b07 g_usleep (libglib-2.0.so.0 + 0x7eb07)
                #3  0x000055be061d4bda call_rcu_thread (qemu-kvm + 0x9cebda)
                #4  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #5  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128320:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128222:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128217:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128213:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055be061c835f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055be05cde60b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055be05f88de1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128328:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128317:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128340:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128330:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128318:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128332:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128336:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128329:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128333:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128341:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128331:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128342:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128345:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128339:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128412:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128319:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128327:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128334:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128335:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128337:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128343:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128347:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128324:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128346:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128338:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128325:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 128326:
                #0  0x00007fc902e9c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007fc902e9eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055be061c84fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055be061c84a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055be061e32e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055be061c8daa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007fc902e9f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007fc902e3f450 __clone3 (libc.so.6 + 0x3f450)
                ELF object binary architecture: AMD x86-64
Comment 17 aihua liang 2023-06-20 08:20:45 UTC 
Test on qemu-kvm-8.0.0-5.el9 with virtio_blk and virtio_scsi for 200 times, all pass.
(200/200) repeat200..Host_RHEL.m9.u3.ovmf.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.3.0.x86_64.io-github-autotest-qemu.blockdev_mirror_after_block_error.q35: PASS
(200/200) repeat200.Host_RHEL.m9.u3.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.RHEL.9.3.0.x86_64.io-github-autotest-qemu.blockdev_mirror_after_block_error.q35: PASS

So will close it as currentrelease.