RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2155112 - Qemu coredump after do snapshot of mirrored top image and its converted base image(iothread enabled)
Summary: Qemu coredump after do snapshot of mirrored top image and its converted base ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Stefano Garzarella
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-12-20 02:49 UTC by aihua liang
Modified: 2023-05-09 07:48 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-7.2.0-5.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-09 07:20:55 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Have external_snapshot_prepare() lock the overlay’s AioContext (1.77 KB, patch)
2023-01-02 15:33 UTC, Hanna Czenczek 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Gitlab redhat/centos-stream/src qemu-kvm merge_requests 135 0 None opened Draft: block: Simplify drain to prevent QEMU from crashing during snapshot 2023-01-13 11:09:30 UTC
Red Hat Issue Tracker RHELPLAN-142924 0 None None None 2022-12-20 02:51:19 UTC
Red Hat Product Errata RHSA-2023:2162 0 None None None 2023-05-09 07:21:42 UTC

Description aihua liang 2022-12-20 02:49:30 UTC 
Description of problem:
Qemu coredump after do snapshot on mirrored top image and its converted base image(iothread enabled)

Version-Release number of selected component (if applicable):
kernel version:5.14.0-212.el9.x86_64
qemu-kvm version:qemu-kvm-7.2.0-1.el9

How reproducible:
 100% by auto

Steps to Reproduce:
1.Start guest with qemu cmd:
  /usr/libexec/qemu-kvm \
     -S  \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
     -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
     -blockdev node-name=file_ovmf_vars,driver=file,filename=/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd,auto-read-only=on,discard=unmap \
     -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
     -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
     -nodefaults \
     -m 30720 \
     -object '{"qom-type": "memory-backend-ram", "size": 32212254720, "id": "mem-machine_mem"}'  \
     -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
     -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
     -chardev socket,path=/var/tmp/avocado_pzd6v87x/monitor-qmpmonitor1-20221219-011444-yoM1Fk2J,wait=off,server=on,id=qmp_id_qmpmonitor1  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,path=/var/tmp/avocado_pzd6v87x/monitor-catch_monitor-20221219-011444-yoM1Fk2J,wait=off,server=on,id=qmp_id_catch_monitor  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device pvpanic,ioport=0x505,id=iddutSce \
     -chardev socket,path=/var/tmp/avocado_pzd6v87x/serial-serial0-20221219-011444-yoM1Fk2J,wait=off,server=on,id=chardev_serial0 \
-device isa-serial,id=serial0,chardev=chardev_serial0  \
     -chardev socket,id=seabioslog_id_20221219-011444-yoM1Fk2J,path=/var/tmp/avocado_pzd6v87x/seabios-20221219-011444-yoM1Fk2J,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20221219-011444-yoM1Fk2J,iobase=0x402 \
     -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel920-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1",      -blockdev '{"node-name": "file_data1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/root/avocado/data/avocado-vt/data1.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_data1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_data1"}' \
     -device '{"driver": "scsi-hd", "id": "data1", "drive": "drive_data1", "write-cache": "on"}' \
     -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
     -device virtio-net-pci,mac=9a:36:b9:70:1e:73,id=idGMJM6z,netdev=idDwXrrb,bus=pcie-root-port-3,addr=0x0  \
     -netdev tap,id=idDwXrrb,vhost=on,vhostfd=16,fd=9  \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -chardev socket,id=char_vtpm_avocado-vt-vm1_tpm0,path=/root/avocado/data/avocado-vt/swtpm/avocado-vt-vm1_tpm0_swtpm.sock \
     -tpmdev emulator,chardev=char_vtpm_avocado-vt-vm1_tpm0,id=emulator_vtpm_avocado-vt-vm1_tpm0 \
     -device tpm-crb,id=tpm-crb_vtpm_avocado-vt-vm1_tpm0,tpmdev=emulator_vtpm_avocado-vt-vm1_tpm0 \
     -enable-kvm \
     -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5

2.Continue guest, after guest is up, write some data in data disk.
  {'execute': 'cont', 'id': 'eQ1yWMQI'}
  (guest)#parted -s "/dev/sdb" mkpart primary 0M 2048.0M
         #yes|mkfs.ext4 -F '/dev/sdb1'
         #mkdir /mnt/sdb1
         #mount -t ext4 /dev/sdb1 /mnt/sdb1
         #dd if=/dev/urandom of=/mnt/sdb1/qGmy bs=1M count=10 oflag=direct
         #md5sum /mnt/sdb1/qGmy > /mnt/sdb1/qGmy.md5 && sync
         
3.Create snapshot of data disk, and add it.
  #qemu-img create -f qcow2 -b /root/avocado/data/avocado-vt/data1.qcow2 -F qcow2 /root/avocado/data/avocado-vt/data1sn.qcow2 2G
  {'execute': 'blockdev-add', 'arguments': {'node-name': 'file_data1sn', 'driver': 'file', 'auto-read-only': True, 'discard': 'unmap', 'aio': 'threads', 'filename': '/root/avocado/data/avocado-vt/data1sn.qcow2', 'cache': {'direct': True, 'no-flush': False}}, 'id': 'NEUuPONk'}
  {'execute': 'blockdev-add', 'arguments': {'node-name': 'drive_data1sn', 'driver': 'qcow2', 'read-only': False, 'cache': {'direct': True, 'no-flush': False}, 'file': 'file_data1sn', 'backing': None}, 'id': '3b56DU3G'}

4.Do snapshot of data to snapshot file.
  {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_data1', 'overlay': 'drive_data1sn'}, 'id': 'nyuVOuu2'}

5. Write some new data on snapshot file.
  (guest)#dd if=/dev/urandom of=/mnt/sdb1/msk4 bs=1M count=10 oflag=direct
         #md5sum /mnt/sdb1/msk4 > /mnt/sdb1/msk4.md5 && sync

6. Create a target convert image.
   #qemu-img create -f qcow2 /root/avocado/data/avocado-vt/convert1.qcow2 2G

7. Create mirror target image online.
   {'execute': 'blockdev-create', 'arguments': {'options': {'driver': 'file', 'filename': '/root/avocado/data/avocado-vt/convert1sn.qcow2', 'size': 2147483648}, 'job-id': 'file_convert1sn'}, 'id': 'yROlWk2R'}
   {'execute': 'job-dismiss', 'arguments': {'id': 'file_convert1sn'}, 'id': 'nvjJ6Ps8'}
   {'execute': 'blockdev-add', 'arguments': {'node-name': 'file_convert1sn', 'driver': 'file', 'filename': '/root/avocado/data/avocado-vt/convert1sn.qcow2', 'aio': 'threads', 'auto-read-only': True, 'discard': 'unmap'}, 'id': '6LsNR4hA'}
   {'execute': 'blockdev-create', 'arguments': {'options': {'driver': 'qcow2', 'file': 'file_convert1sn', 'size': 2147483648, 'backing-fmt': 'qcow2', 'backing-file': '/root/avocado/data/avocado-vt/convert1.qcow2'}, 'job-id': 'drive_convert1sn'}, 'id': 'DLWByep4'}
   {'execute': 'job-dismiss', 'arguments': {'id': 'drive_convert1sn'}, 'id': 'paf9Hnni'}
   {'execute': 'blockdev-add', 'arguments': {'node-name': 'drive_convert1sn', 'driver': 'qcow2', 'backing': None, 'file': 'file_convert1sn', 'read-only': False}, 'id': 'rNswMbRV'}

7. Mirror from src to target with sync "top".
   {'execute': 'blockdev-mirror', 'arguments': {'sync': 'top', 'device': 'drive_data1sn', 'target': 'drive_convert1sn', 'job-id': 'drive_data1sn_JJv9'}, 'id': 'dDgd6Dei'}

8. After mirror complted, convert image from src base to target base.
   #qemu-img convert -f qcow2 -O qcow2 /root/avocado/data/avocado-vt/data1.qcow2 /root/avocado/data/avocado-vt/convert1.qcow2

9. Add the converted target base image
   {'execute': 'blockdev-add', 'arguments': {'node-name': 'file_convert1', 'driver': 'file', 'auto-read-only': True, 'discard': 'unmap', 'aio': 'threads', 'filename': '/root/avocado/data/avocado-vt/convert1.qcow2', 'cache': {'direct': True, 'no-flush': False}}, 'id': 'Fdy1STFP'}
   {'execute': 'blockdev-add', 'arguments': {'node-name': 'drive_convert1', 'driver': 'qcow2', 'read-only': False, 'cache': {'direct': True, 'no-flush': False}, 'file': 'file_convert1'}, 'id': 'iDUdUtrr'}

10. Do snapshot from target top to target base.
    {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_convert1', 'overlay': 'drive_convert1sn'}, 'id': 'oOm0J5Wt'}

Actual results:
After step10, qemu coredump with info:
 qemu: qemu_mutex_unlock_impl: Operation not permitted
 /tmp/aexpect_dM1RN4TM/aexpect-4u3euwsm.sh: line 1: 115066 Aborted                 (core dumped) MALLOC_PERTURB_=1 /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code -blockdev node-name=file_ovmf_vars,driver=file,filename=/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd,auto-read-only=on,discard=unmap -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 30720 -object '{"qom-type": "memory-backend-ram", "size": 32212254720, "id": "mem-machine_mem"}' -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2 ....

Cordump info:
#coredumpctl debug 115066
 Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-0.slice
       Session: 1
     Owner UID: 0 (root)
       Boot ID: 5df88b14112e4a4fb377b4ce7912ae53
    Machine ID: 8ba0e28e086e4a6b850e4904917fe912
      Hostname: dell-per440-08.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.5df88b14112e4a4fb377b4ce7912ae53.115066.1671430536000000.zst (present)
  Size on Disk: 569.6M
       Message: Process 115066 (qemu-kvm) of user 0 dumped core.
                
                Stack trace of thread 115066:
                #0  0x00007ff9cb8a154c __pthread_kill_implementation (libc.so.6 + 0xa154c)
                #1  0x00007ff9cb854d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007ff9cb8287f3 abort (libc.so.6 + 0x287f3)
                #3  0x00005573ad75f472 qemu_mutex_unlock_impl (qemu-kvm + 0x9bd472)
                #4  0x00005573ad5bcd9c bdrv_parent_drained_begin_single (qemu-kvm + 0x81ad9c)
                #5  0x00005573ad57cfc8 bdrv_replace_child_noperm (qemu-kvm + 0x7dafc8)
                #6  0x00005573ad57f4cb bdrv_attach_child_common (qemu-kvm + 0x7dd4cb)
                #7  0x00005573ad57fc8a bdrv_attach_child_noperm (qemu-kvm + 0x7ddc8a)
                #8  0x00005573ad5861f3 bdrv_append (qemu-kvm + 0x7e41f3)
                #9  0x00005573ad56ccd1 external_snapshot_prepare (qemu-kvm + 0x7cacd1)
                #10 0x00005573ad56b56d qmp_transaction (qemu-kvm + 0x7c956d)
                #11 0x00005573ad67fca6 qmp_marshal_blockdev_snapshot (qemu-kvm + 0x8ddca6)
                #12 0x00005573ad74f872 do_qmp_dispatch_bh (qemu-kvm + 0x9ad872)
                #13 0x00005573ad75b871 aio_dispatch (qemu-kvm + 0x9b9871)
                #14 0x00005573ad775f42 aio_ctx_dispatch (qemu-kvm + 0x9d3f42)
                #15 0x00007ff9cbed9e2f g_main_context_dispatch (libglib-2.0.so.0 + 0x54e2f)
                #16 0x00005573ad777e44 main_loop_wait (qemu-kvm + 0x9d5e44)
                #17 0x00005573ad281597 qemu_main_loop (qemu-kvm + 0x4df597)
                #18 0x00005573ad10792a qemu_default_main (qemu-kvm + 0x36592a)
                #19 0x00007ff9cb83feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #20 0x00007ff9cb83ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #21 0x00005573ad107085 _start (qemu-kvm + 0x365085)
                
                Stack trace of thread 115067:
                #0  0x00007ff9cb83ee5d syscall (libc.so.6 + 0x3ee5d)
                #1  0x00005573ad75ffbf qemu_event_wait (qemu-kvm + 0x9bdfbf)
                #2  0x00005573ad76c0f5 call_rcu_thread (qemu-kvm + 0x9ca0f5)
                #3  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #4  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #5  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115073:
                #0  0x00007ff9cb9429bf __poll (libc.so.6 + 0x1429bf)
                #1  0x00007ff9cbf2e49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c)
                #2  0x00007ff9cbed9483 g_main_loop_run (libglib-2.0.so.0 + 0x54483)
                #3  0x00005573ad57454f iothread_run (qemu-kvm + 0x7d254f)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115074:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115078:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115079:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115076:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115082:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115085:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x00005573ad75f81f qemu_cond_wait_impl (qemu-kvm + 0x9bd81f)
                #3  0x00005573ad13dca6 vnc_worker_thread (qemu-kvm + 0x39bca6)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115081:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115077:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115075:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115083:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115166:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115147:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115080:
                #0  0x00007ff9cb83ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x00005573ad51a497 kvm_vcpu_ioctl (qemu-kvm + 0x778497)
                #2  0x00005573ad51f981 kvm_cpu_exec (qemu-kvm + 0x77d981)
                #3  0x00005573ad521e1a kvm_vcpu_thread_fn (qemu-kvm + 0x77fe1a)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115168:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115170:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115068:
                #0  0x00007ff9cb942abe ppoll (libc.so.6 + 0x142abe)
                #1  0x00005573ad75cd5e fdmon_poll_wait (qemu-kvm + 0x9bad5e)
                #2  0x00005573ad75bf9e aio_poll (qemu-kvm + 0x9b9f9e)
                #3  0x00005573ad574532 iothread_run (qemu-kvm + 0x7d2532)
                #4  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #5  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115210:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115174:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115179:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115171:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115178:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115177:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115167:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115169:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115146:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115173:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115172:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115175:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 115176:
                #0  0x00007ff9cb89c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007ff9cb89eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x00005573ad75f9bc qemu_cond_timedwait_ts (qemu-kvm + 0x9bd9bc)
                #3  0x00005573ad75f960 qemu_cond_timedwait_impl (qemu-kvm + 0x9bd960)
                #4  0x00005573ad77a727 worker_thread (qemu-kvm + 0x9d8727)
                #5  0x00005573ad76026a qemu_thread_start (qemu-kvm + 0x9be26a)
                #6  0x00007ff9cb89f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007ff9cb83f450 __clone3 (libc.so.6 + 0x3f450)
                ELF object binary architecture: AMD x86-64

Expected results:
 Snapshot from target base to top can executed successfully.

Additional info:
 Only when iothread enabled can hit this issue, and both virtio_blk and virtio_scsi hit this issue.
Comment 2 aihua liang 2022-12-20 03:11:51 UTC 
Test with qemu-kvm-7.1.0-7.el9, not hit this issue.
So it's a regression issue, set the keyword.
Comment 3 Klaus Heinrich Kiwi 2022-12-21 14:16:08 UTC 
We have low coverage during the holidays, so Stefano will take a quick look, and try to reassign it to be addressed early January..

Paolo and Stefan fyi
Comment 4 Stefano Garzarella 2022-12-21 16:54:33 UTC 
I'm able to reproduce with the upstream QEMU v7.2.0
This issue seems related to the following commit:

commit ace5a161ea1c09d8eaa8b2a717528457dc924e83
Author: Hanna Reitz <hreitz>
Date:   Mon Nov 7 16:13:21 2022 +0100

    block: Start/end drain on correct AioContext
    
    bdrv_parent_drained_{begin,end}_single() are supposed to operate on the
    parent, not on the child, so they should not attempt to get the context
    to poll from the child but the parent instead.  BDRV_POLL_WHILE(c->bs)
    does get the context from the child, so we should replace it with
    AIO_WAIT_WHILE() on the parent's context instead.
    
    This problem becomes apparent when bdrv_replace_child_noperm() invokes
    bdrv_parent_drained_end_single() after removing a child from a subgraph
    that is in an I/O thread.  By the time bdrv_parent_drained_end_single()
    is called, child->bs is NULL, and so BDRV_POLL_WHILE(c->bs, ...) will
    poll the main loop instead of the I/O thread; but anything that
    bdrv_parent_drained_end_single_no_poll() may have scheduled is going to
    want to run in the I/O thread, but because we poll the main loop, the
    I/O thread is never unpaused, and nothing is run, resulting in a
    deadlock.
    
    Closes: https://gitlab.com/qemu-project/qemu/-/issues/1215
    Reviewed-by: Kevin Wolf <kwolf>
    Signed-off-by: Hanna Reitz <hreitz>
    Message-Id: <20221107151321.211175-4-hreitz>
    Signed-off-by: Kevin Wolf <kwolf>
    
Reverting it fixes this issue, but I'm not sure if it is the right thing to do, since Hanna tried to fix another issue.

In addition, I tried the latest QEMU master branch (700ce3b1bb52da4acbbf1ad8f6256baaf52c7953) and the issue seems to be solved.
It contains some reworks made by Kevin around bdrv_parent_drained_begin_single() for QEMU v8.0.
I'm not sure if we can backport all of them.

I would like some advice from Hanna and Kevin before proceeding. They are on PTO and I will be in PTO from Dec 24 to Jan 8, but I think we could solve this problem as soon as we get back.
Comment 5 Hanna Czenczek 2023-01-02 15:30:51 UTC 
Yes, that commit is a bug fix, so I wouldn’t just revert it.

Kevin’s series (“block: Simplify drain>") does revert it, because it happens to fix the issue in another way.  Backporting Kevin’s series doesn’t seem like the worst idea to me.


On the issue at hand: I find the description in commit 0 not quite correct, specifically “After mirror complted” in step 8.  If I let the mirror job complete, the guest device will point to drive_convert1sn, and blockdev-snapshot (step 10) will fail (“The overlay is already in use”).  Steps 8 through 10 must be run before the mirror job has completed.

(It would also be wrong to let mirror complete before step 10, because drive_convert1sn is lacking the base image before that point, so having the mirror complete would mean the guest would no longer see the data in the base image.)

I think the problem is that callers of bdrv_append() should (before Kevin’s series) lock all AioContexts that are involved in the operation.  Right now, it only locks the base image’s context, but then we quiesce the overlay’s parent (namely the mirror job), and I believe it is correct to poll the mirror job’s context for this (which is what my patch has changed).  But for this to work, the caller (external_snapshot_prepare()) should have locked the mirror job’s context, i.e. the overlay image’s context, which it doesn’t do.

I think this all becomes moot with Kevin’s series (thankfully), but if we need a quick fix, I think it should be sufficient to have external_snapshot_prepare() lock new_bs’s AioContext, too, if it differs from old_bs’s.;  I’ll attach a diff to that effect.  (It seems to fix the issue for me at least...)
Comment 6 Hanna Czenczek 2023-01-02 15:32:35 UTC 
(Sorry for the typo in comment 5, I meant “description in comment 0”, not “description in commit 0”.)
Comment 7 Hanna Czenczek 2023-01-02 15:33:19 UTC 
Created attachment 1935332 [details]
Have external_snapshot_prepare() lock the overlay’s AioContext
Comment 8 Vivek Goyal 2023-01-04 19:35:16 UTC 
So what's next? Hanna, should QE test again with your patch and see if it fixes the issue?

I feel that a quick fix probably is better. And then Kevin's series could go on top reverting the fix. So that we don't rely on having to backport kevin's patch series to fix this issue. (Kevin's patches are yet to get merged upstream, IIUC).
Comment 9 Hanna Czenczek 2023-01-05 08:00:08 UTC 
I’d prefer to first wait on Stefano’s and Kevin’s opinions before continuing.  Kevin’s patches are upstream in 8.0.
Comment 17 aihua liang 2023-01-18 02:42:36 UTC 
Test by auto, it works on qemu-kvm-7.2.0-5.el9.
(1/2) Host_RHEL.m9.u2.ovmf.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.2.0.x86_64.io-github-autotest-qemu.blockdev_mirror_sync_top.q35: PASS (136.45 s)
(2/2) Host_RHEL.m9.u2.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.RHEL.9.2.0.x86_64.io-github-autotest-qemu.blockdev_mirror_sync_top.q35: PASS (153.05 s)
Comment 18 Yanan Fu 2023-01-19 05:24:23 UTC 
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.
Comment 22 aihua liang 2023-01-20 01:45:10 UTC 
As comment 17 and comment 18, set bug's status to "VERIFIED".
Comment 23 aihua liang 2023-02-03 07:08:45 UTC 
Hi,Stefano

  When running tests on qemu-kvm-7.2.0-6.el9, I still hit this issue, with reproduce ratio lower than 20%.(I run the case 50 times, 7 times I hit this issue)

Coredump info:
  Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-9.scope
          Unit: session-9.scope
         Slice: user-0.slice
       Session: 9
     Owner UID: 0 (root)
       Boot ID: 52c869c21ef64de49877ac0eed7aeb06
    Machine ID: 3919555703fd4043b7f3cc2611ad4d18
      Hostname: dell-per740xd-01.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.52c869c21ef64de49877ac0eed7aeb06.363458.1675305507000000.zst (present)
  Size on Disk: 304.5M
       Message: Process 363458 (qemu-kvm) of user 0 dumped core.
                
                Stack trace of thread 363458:
                #0  0x00007f03026a154c __pthread_kill_implementation (libc.so.6 + 0xa154c)
                #1  0x00007f0302654d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007f03026287f3 abort (libc.so.6 + 0x287f3)
                #3  0x000055d3bc22dff2 qemu_mutex_unlock_impl (qemu-kvm + 0x9bdff2)
                #4  0x000055d3bc08cda7 bdrv_do_drained_begin (qemu-kvm + 0x81cda7)
                #5  0x000055d3bc055e1e bdrv_replace_node_noperm (qemu-kvm + 0x7e5e1e)
                #6  0x000055d3bc055c92 bdrv_append (qemu-kvm + 0x7e5c92)
                #7  0x000055d3bc03c62c external_snapshot_prepare (qemu-kvm + 0x7cc62c)
                #8  0x000055d3bc03aedd qmp_transaction (qemu-kvm + 0x7caedd)
                #9  0x000055d3bc14e826 qmp_marshal_blockdev_snapshot (qemu-kvm + 0x8de826)
                #10 0x000055d3bc21e3f2 do_qmp_dispatch_bh (qemu-kvm + 0x9ae3f2)
                #11 0x000055d3bc22a3f1 aio_dispatch (qemu-kvm + 0x9ba3f1)
                #12 0x000055d3bc2450a2 aio_ctx_dispatch (qemu-kvm + 0x9d50a2)
                #13 0x00007f0302d1ae2f g_main_context_dispatch (libglib-2.0.so.0 + 0x54e2f)
                #14 0x000055d3bc2469c4 main_loop_wait (qemu-kvm + 0x9d69c4)
                #15 0x000055d3bbd4f8e7 qemu_main_loop (qemu-kvm + 0x4df8e7)
                #16 0x000055d3bbbd592a qemu_default_main (qemu-kvm + 0x36592a)
                #17 0x00007f030263feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #18 0x00007f030263ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #19 0x000055d3bbbd5085 _start (qemu-kvm + 0x365085)
                
                Stack trace of thread 363465:
                #0  0x00007f03027429bf __poll (libc.so.6 + 0x1429bf)
                #1  0x00007f0302d6f49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c)
                #2  0x00007f0302d1a483 g_main_loop_run (libglib-2.0.so.0 + 0x54483)
                #3  0x000055d3bc043e2f iothread_run (qemu-kvm + 0x7d3e2f)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363470:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363468:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363476:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363570:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363460:
                #0  0x00007f0302742abe ppoll (libc.so.6 + 0x142abe)
                #1  0x000055d3bc22b8de fdmon_poll_wait (qemu-kvm + 0x9bb8de)
                #2  0x000055d3bc22ab1e aio_poll (qemu-kvm + 0x9bab1e)
                #3  0x000055d3bc043e12 iothread_run (qemu-kvm + 0x7d3e12)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363548:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363466:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363572:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363571:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363459:
                #0  0x00007f030263ee5d syscall (libc.so.6 + 0x3ee5d)
                #1  0x000055d3bc22eb3f qemu_event_wait (qemu-kvm + 0x9beb3f)
                #2  0x000055d3bc23ac75 call_rcu_thread (qemu-kvm + 0x9cac75)
                #3  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #4  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #5  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363475:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363472:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363580:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363477:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363577:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363474:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363584:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363573:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363479:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055d3bc22e39f qemu_cond_wait_impl (qemu-kvm + 0x9be39f)
                #3  0x000055d3bbc0bc76 vnc_worker_thread (qemu-kvm + 0x39bc76)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363583:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363473:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363469:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363581:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363619:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363587:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363586:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363578:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363471:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363579:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363585:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363582:
                #0  0x00007f030269c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f030269eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055d3bc22e53c qemu_cond_timedwait_ts (qemu-kvm + 0x9be53c)
                #3  0x000055d3bc22e4e0 qemu_cond_timedwait_impl (qemu-kvm + 0x9be4e0)
                #4  0x000055d3bc2492a7 worker_thread (qemu-kvm + 0x9d92a7)
                #5  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #6  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 363467:
                #0  0x00007f030263ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055d3bbfe998b kvm_vcpu_ioctl (qemu-kvm + 0x77998b)
                #2  0x000055d3bbfef191 kvm_cpu_exec (qemu-kvm + 0x77f191)
                #3  0x000055d3bbff178a kvm_vcpu_thread_fn (qemu-kvm + 0x78178a)
                #4  0x000055d3bc22edea qemu_thread_start (qemu-kvm + 0x9bedea)
                #5  0x00007f030269f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f030263f450 __clone3 (libc.so.6 + 0x3f450)
                ELF object binary architecture: AMD x86-64 

 I tried with qemu-kvm-7.2.0-5.el9, and still reproduce it with a ratio: 4/50

 Then I run the case on qemu-kvm-7.1.0-7.el9 for 100 times, and all tests pass.

 So from the test result, we can see that the code fix reduces the reproduce ratio to a low level, but not a complete fix.

 As it's still a regression bug but with a lower reproduce ratio, I'm not sure if we need to file a new bug to track, or just use this existing one is ok?

BR,
Aliang
Comment 25 Stefano Garzarella 2023-02-08 13:51:56 UTC 
(In reply to aihua liang from comment #23)
> 
>  So from the test result, we can see that the code fix reduces the reproduce
> ratio to a low level, but not a complete fix.

Yep, Hanna and Kevin suggested on IRC to take a look at locking in bdrv_append().
I'll do in the next days.

> 
>  As it's still a regression bug but with a lower reproduce ratio, I'm not
> sure if we need to file a new bug to track, or just use this existing one is
> ok?

Just to update this BZ, we agreed to create a new BZ since we need to fix qemu upstream and then backport the patch downstream.

The new BZ is BZ2168209. We can continue to discuss there.
Comment 27 errata-xmlrpc 2023-05-09 07:20:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2162
Note You need to log in before you can comment on or make changes to this bug.