Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2186441

Summary: qemu: qemu_mutex_unlock_impl: Operation not permitted(iothread enable)
Product: Red Hat Enterprise Linux 9 Reporter: aihua liang <aliang>
Component: qemu-kvmAssignee: Kevin Wolf <kwolf>
qemu-kvm sub component: Incremental Live Backup QA Contact: aihua liang <aliang>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: high    
Priority: high CC: coli, jinzhao, juzhang, kwolf, vgoyal, virt-maint, zhguo
Version: 9.3Keywords: CustomerScenariosInitiative, Regression, Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-18 10:59:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description aihua liang 2023-04-13 09:49:02 UTC
Description of problem:
qemu: qemu_mutex_unlock_impl: Operation not permitted(iothread enable)

Version-Release number of selected component (if applicable):
kernel version:5.14.0-290.kpq1.el9.x86_64
qemu-kvm version:qemu-kvm-8.0.0-0.rc1.el9.candidate

How reproducible:
5/5

Steps to Reproduce:
1.Start src guest with qemu cmdline:
  /usr/libexec/qemu-kvm \
     -S  \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel930-64-virtio_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"size": 32212254720, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
     -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
     -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
     -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20230413-050654-SEkZfBqX,wait=off,id=qmp_id_qmpmonitor1  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,server=on,path=/var/tmp/monitor-catch_monitor-20230413-050654-SEkZfBqX,wait=off,id=qmp_id_catch_monitor  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idnRLXj0"}' \
     -chardev socket,server=on,path=/var/tmp/serial-serial0-20230413-050654-SEkZfBqX,wait=off,id=chardev_serial0 \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230413-050654-SEkZfBqX,path=/var/tmp/seabios-20230413-050654-SEkZfBqX,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230413-050654-SEkZfBqX,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel930-64-virtio.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 0, "write-cache": "on", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_data", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/root/avocado/data/avocado-vt/data.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -object '{"qom-type": "iothread", "id": "iothread1"}' \
     -blockdev '{"node-name": "drive_data", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_data"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-blk-pci", "id": "data", "drive": "drive_data", "bootindex": 1, "write-cache": "on", "bus": "pcie-root-port-3", "addr": "0x0", "iothread": "iothread1"}' \
     -device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:51:97:3a:13:46", "id": "idaB6X7z", "netdev": "idj45xY4", "bus": "pcie-root-port-4", "addr": "0x0"}'  \
     -netdev tap,id=idj45xY4,vhost=on  \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 6}' \
     -monitor stdio \

2.Continue vm, after it's up, in guest, format data disk and create a new file
  (qemu)c
  (guest)#parted -s "/dev/vdb" mklabel msdos
         #parted -s "/dev/vdb" mkpart primary 0M 2048.0M
         #yes|mkfs.ext4 -F '/dev/vdb1'
         #mount -t ext4 /dev/vdb1 /mnt/vdb1
         #dd if=/dev/urandom of=/mnt/vdb1/u6Lp bs=1M count=10 oflag=direct
         #md5sum /mnt/vdb1/u6Lp > /mnt/vdb1/u6Lp.md5 && sync

3.Start dst vm with qemu cmdline:
  /usr/libexec/qemu-kvm \
     -S  \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel930-64-virtio_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"size": 32212254720, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
     -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
     -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
     -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20230413-050654-SEkZfBqX,wait=off,id=qmp_id_qmpmonitor1  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,server=on,path=/var/tmp/monitor-catch_monitor-20230413-050654-SEkZfBqX,wait=off,id=qmp_id_catch_monitor  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idnRLXj0"}' \
     -chardev socket,server=on,path=/var/tmp/serial-serial0-20230413-050654-SEkZfBqX,wait=off,id=chardev_serial0 \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230413-050752-GYeBvUEV,path=/var/tmp/seabios-20230413-050752-GYeBvUEV,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230413-050752-GYeBvUEV,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel930-64-virtio.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 0, "write-cache": "on", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_data", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/root/avocado/data/avocado-vt/data.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -object '{"qom-type": "iothread", "id": "iothread1"}' \
     -blockdev '{"node-name": "drive_data", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_data"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-blk-pci", "id": "data", "drive": "drive_data", "bootindex": 1, "write-cache": "on", "bus": "pcie-root-port-3", "addr": "0x0", "iothread": "iothread1"}' \
     -device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:51:97:3a:13:46", "id": "idH37UNL", "netdev": "idRruXsx", "bus": "pcie-root-port-4", "addr": "0x0"}'  \
     -netdev tap,id=idRruXsx,vhost=on  \
     -vnc :1  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 6}' \
     -incoming defer \
     -monitor stdio \

4.In src, create target backup node
  {"execute": "blockdev-create", "arguments": {"options": {"driver": "file", "filename": "/root/avocado/data/avocado-vt/base.qcow2", "size": 2147483648}, "job-id": "file_base"}, "id": "1er6syhm"}
  {"execute": "job-dismiss", "arguments": {"id": "file_base"}, "id": "8q5Eo5qm"}
  {"execute": "blockdev-add", "arguments": {"node-name": "file_base", "driver": "file", "filename": "/root/avocado/data/avocado-vt/base.qcow2", "aio": "threads", "auto-read-only": true, "discard": "unmap"}, "id": "RaYRu0EE"}
  {"execute": "blockdev-create", "arguments": {"options": {"driver": "qcow2", "file": "file_base", "size": 2147483648}, "job-id": "drive_base"}, "id": "zn5D27y0"}
  {"execute": "job-dismiss", "arguments": {"id": "drive_base"}, "id": "PWFf9V2s"}
  {"execute": "blockdev-add", "arguments": {"node-name": "drive_base", "driver": "qcow2", "file": "file_base", "read-only": false}, "id": "IlUFekXf"}

5.In src, do full backup and add a bitmap
  {"execute": "transaction", "arguments": {"actions": [{"type": "blockdev-backup", "data": {"device": "drive_data", "target": "drive_base", "job-id": "drive_data_1TrV", "sync": "full", "speed": 0, "compress": false, "auto-finalize": true, "auto-dismiss": true, "on-source-error": "report", "on-target-error": "report"}}, {"type": "block-dirty-bitmap-add", "data": {"node": "drive_data", "name": "bitmap_data"}}]}, "id": "hLS23EB5"}

6.After backup done, in src guest, create a new file
  (src guest)#dd if=/dev/urandom of=/mnt/vdb1/mlNx bs=1M count=10 oflag=direct
             #md5sum /mnt/vdb1/mlNx > /mnt/vdb1/mlNx.md5 && sync

7.In src, disable bitmap
  {"execute": "block-dirty-bitmap-disable", "arguments": {"node": "drive_data", "name": "bitmap_data"}, "id": "dXlh5Cm9"}

8.In src, set migraiton capabilities
  {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "pause-before-switchover"}]}, "id": "Ksu5d8sM"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "events"}]}, "id": "YqWcEVon"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "dirty-bitmaps"}]}, "id": "dG1Ier5x"}

9.In dst, set migraiton capabilities and set migrate-incoming
  {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "late-block-activate"}]}, "id": "67C8AC3j"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "events"}]}, "id": "YTZLJMTT"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "dirty-bitmaps"}]}, "id": "dzWpKBhh"}
{"execute": "migrate-incoming", "arguments": {"uri": "tcp:[::]:5531"}, "id": "Xu7H65iD"}

10.Migrate from src to dst.
  {"execute": "migrate", "arguments": {"uri": "tcp:localhost:5531", "blk": false, "inc": false}, "id": "4WC6sQfJ"}
{"timestamp": {"seconds": 1681445417, "microseconds": 781743}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}, "id": "4WC6sQfJ"}
{"timestamp": {"seconds": 1681445418, "microseconds": 12917}, "event": "MIGRATION_PASS", "data": {"pass": 1}}


Actual results:
After step10, qemu coredump
Src qmp monitor output:
 {"timestamp": {"seconds": 1681445417, "microseconds": 781743}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}, "id": "4WC6sQfJ"}
{"timestamp": {"seconds": 1681445418, "microseconds": 12917}, "event": "MIGRATION_PASS", "data": {"pass": 1}}

Dst qmp monitor output:
 {"timestamp": {"seconds": 1681445408, "microseconds": 164487}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}, "id": "Xu7H65iD"}
{"timestamp": {"seconds": 1681445417, "microseconds": 782170}, "event": "MIGRATION", "data": {"status": "active"}}
{"timestamp": {"seconds": 1681445418, "microseconds": 71210}, "event": "MIGRATION", "data": {"status": "failed"}}


Qemu coredump with info:
 qemu: qemu_mutex_unlock_impl: Operation not permitted
src.txt: line 49: 510966 Aborted                 (core dumped) /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' ...

Coredump info:
Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-44.scope
          Unit: session-44.scope
         Slice: user-0.slice
       Session: 44
     Owner UID: 0 (root)
       Boot ID: 3fc872d61fd14fc3ae2cdf1cf487f0f7
    Machine ID: 58bf48c028d34ee69040709abf09d8ec
      Hostname: dell-per440-08.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.3fc872d61fd14fc3ae2cdf1cf487f0f7.510966.1681376197000000.zst (present)
  Size on Disk: 654.9M
       Message: Process 510966 (qemu-kvm) of user 0 dumped core.
                
                Stack trace of thread 511408:
                #0  0x00007fb843ca157c __pthread_kill_implementation (libc.so.6 + 0xa157c)
                #1  0x00007fb843c54d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007fb843c287f3 abort (libc.so.6 + 0x287f3)
                #3  0x000055b9e1a8b112 qemu_mutex_unlock_impl (qemu-kvm + 0x9e8112)
                #4  0x000055b9e18962cd bdrv_poll_co (qemu-kvm + 0x7f32cd)
                #5  0x000055b9e1890e2e bdrv_nb_sectors (qemu-kvm + 0x7ede2e)
                #6  0x000055b9e15c50bb add_bitmaps_to_list (qemu-kvm + 0x5220bb)
                #7  0x000055b9e15c3a34 dirty_bitmap_save_setup (qemu-kvm + 0x520a34)
                #8  0x000055b9e15e35f2 qemu_savevm_state_setup (qemu-kvm + 0x5405f2)
                #9  0x000055b9e15d5053 migration_thread (qemu-kvm + 0x532053)
                #10 0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #11 0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #12 0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510969:
                #0  0x00007fb843d42a6e ppoll (libc.so.6 + 0x142a6e)
                #1  0x000055b9e1a88bee fdmon_poll_wait (qemu-kvm + 0x9e5bee)
                #2  0x000055b9e1a883c6 aio_poll (qemu-kvm + 0x9e53c6)
                #3  0x000055b9e188eee2 iothread_run (qemu-kvm + 0x7ebee2)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510966:
                #0  0x00007fb843c9c590 __GI___lll_lock_wait (libc.so.6 + 0x9c590)
                #1  0x00007fb843ca2c52 __pthread_mutex_lock.5 (libc.so.6 + 0xa2c52)
                #2  0x000055b9e1a8af6f qemu_mutex_lock_impl (qemu-kvm + 0x9e7f6f)
                #3  0x000055b9e1aa419d main_loop_wait (qemu-kvm + 0xa0119d)
                #4  0x000055b9e15a0c47 qemu_main_loop (qemu-kvm + 0x4fdc47)
                #5  0x000055b9e14169ea qemu_default_main (qemu-kvm + 0x3739ea)
                #6  0x00007fb843c3feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #7  0x00007fb843c3ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #8  0x000055b9e1416145 _start (qemu-kvm + 0x373145)
                
                Stack trace of thread 510981:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510983:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510978:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510975:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510967:
                #0  0x00007fb843d13945 clock_nanosleep.5 (libc.so.6 + 0x113945)
                #1  0x00007fb843d184f7 __nanosleep (libc.so.6 + 0x1184f7)
                #2  0x00007fb844303b07 g_usleep (libglib-2.0.so.0 + 0x7eb07)
                #3  0x000055b9e1a981ca call_rcu_thread (qemu-kvm + 0x9f51ca)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510982:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510976:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510979:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510968:
                #0  0x00007fb843d42a6e ppoll (libc.so.6 + 0x142a6e)
                #1  0x000055b9e1a88bee fdmon_poll_wait (qemu-kvm + 0x9e5bee)
                #2  0x000055b9e1a883c6 aio_poll (qemu-kvm + 0x9e53c6)
                #3  0x000055b9e188eee2 iothread_run (qemu-kvm + 0x7ebee2)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511073:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510974:
                #0  0x00007fb843d4296f __poll (libc.so.6 + 0x14296f)
                #1  0x00007fb84432e49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c)
                #2  0x00007fb8442d9483 g_main_loop_run (libglib-2.0.so.0 + 0x54483)
                #3  0x000055b9e188eeff iothread_run (qemu-kvm + 0x7ebeff)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510977:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510986:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9ebd0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9ebd0)
                #2  0x000055b9e1a8b4bf qemu_cond_wait_impl (qemu-kvm + 0x9e84bf)
                #3  0x000055b9e1450116 vnc_worker_thread (qemu-kvm + 0x3ad116)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511077:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511083:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510984:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511146:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511054:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511086:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511144:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511087:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511081:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511082:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511145:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511075:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511154:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511076:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511122:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511151:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511085:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511080:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511124:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511079:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510980:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                ELF object binary architecture: AMD x86-64

Expected results:
Migration can be executed successfully.

Additional info:
 Will provide coredum file later.

Comment 2 aihua liang 2023-04-14 07:27:27 UTC
Not hit this issue on qemu-kvm-7.2.0-14.el9_2 after 20 times test by auto:qemu-kvm-7.2.0-14.el9_2, so it's a regression issue.

Comment 3 aihua liang 2023-04-14 07:49:18 UTC
Both virtio_blk and virtio_scsi hit this issue.

Comment 5 aihua liang 2023-04-14 10:35:09 UTC
Correct the auto script to:blockdev_inc_backup_with_migration

Comment 9 aihua liang 2023-06-18 10:59:23 UTC
Run test case:blockdev_inc_backup_with_migration for 100 times on qemu-kvm-8.0.0-5.el9, all pass.
 (099/100) repeat99.Host_RHEL.m9.u3.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.RHEL.9.3.0.x86_64.io-github-autotest-qemu.blockdev_inc_backup_with_migration.q35: PASS (182.33 s)
 (100/100) repeat100.Host_RHEL.m9.u3.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.RHEL.9.3.0.x86_64.io-github-autotest-qemu.blockdev_inc_backup_with_migration.q35: PASS (185.40 s)
RESULTS    : PASS 100 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /root/avocado/job-results/job-2023-06-18T01.02-ac2e91b/results.html
JOB TIME   : 19388.16 s

And run regression test, all pass.

So will close this bug as currentrelease.