Bug 2186441 - qemu: qemu_mutex_unlock_impl: Operation not permitted(iothread enable)
Summary: qemu: qemu_mutex_unlock_impl: Operation not permitted(iothread enable)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.3
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-13 09:49 UTC by aihua liang
Modified: 2023-07-27 02:44 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-06-18 10:59:23 UTC
Type: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-154612 0 None None None 2023-04-13 16:33:53 UTC

Description aihua liang 2023-04-13 09:49:02 UTC
Description of problem:
qemu: qemu_mutex_unlock_impl: Operation not permitted(iothread enable)

Version-Release number of selected component (if applicable):
kernel version:5.14.0-290.kpq1.el9.x86_64
qemu-kvm version:qemu-kvm-8.0.0-0.rc1.el9.candidate

How reproducible:
5/5

Steps to Reproduce:
1.Start src guest with qemu cmdline:
  /usr/libexec/qemu-kvm \
     -S  \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel930-64-virtio_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"size": 32212254720, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
     -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
     -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
     -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20230413-050654-SEkZfBqX,wait=off,id=qmp_id_qmpmonitor1  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,server=on,path=/var/tmp/monitor-catch_monitor-20230413-050654-SEkZfBqX,wait=off,id=qmp_id_catch_monitor  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idnRLXj0"}' \
     -chardev socket,server=on,path=/var/tmp/serial-serial0-20230413-050654-SEkZfBqX,wait=off,id=chardev_serial0 \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230413-050654-SEkZfBqX,path=/var/tmp/seabios-20230413-050654-SEkZfBqX,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230413-050654-SEkZfBqX,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel930-64-virtio.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 0, "write-cache": "on", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_data", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/root/avocado/data/avocado-vt/data.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -object '{"qom-type": "iothread", "id": "iothread1"}' \
     -blockdev '{"node-name": "drive_data", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_data"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-blk-pci", "id": "data", "drive": "drive_data", "bootindex": 1, "write-cache": "on", "bus": "pcie-root-port-3", "addr": "0x0", "iothread": "iothread1"}' \
     -device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:51:97:3a:13:46", "id": "idaB6X7z", "netdev": "idj45xY4", "bus": "pcie-root-port-4", "addr": "0x0"}'  \
     -netdev tap,id=idj45xY4,vhost=on  \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 6}' \
     -monitor stdio \

2.Continue vm, after it's up, in guest, format data disk and create a new file
  (qemu)c
  (guest)#parted -s "/dev/vdb" mklabel msdos
         #parted -s "/dev/vdb" mkpart primary 0M 2048.0M
         #yes|mkfs.ext4 -F '/dev/vdb1'
         #mount -t ext4 /dev/vdb1 /mnt/vdb1
         #dd if=/dev/urandom of=/mnt/vdb1/u6Lp bs=1M count=10 oflag=direct
         #md5sum /mnt/vdb1/u6Lp > /mnt/vdb1/u6Lp.md5 && sync

3.Start dst vm with qemu cmdline:
  /usr/libexec/qemu-kvm \
     -S  \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel930-64-virtio_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"size": 32212254720, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
     -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
     -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
     -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20230413-050654-SEkZfBqX,wait=off,id=qmp_id_qmpmonitor1  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,server=on,path=/var/tmp/monitor-catch_monitor-20230413-050654-SEkZfBqX,wait=off,id=qmp_id_catch_monitor  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idnRLXj0"}' \
     -chardev socket,server=on,path=/var/tmp/serial-serial0-20230413-050654-SEkZfBqX,wait=off,id=chardev_serial0 \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230413-050752-GYeBvUEV,path=/var/tmp/seabios-20230413-050752-GYeBvUEV,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230413-050752-GYeBvUEV,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel930-64-virtio.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 0, "write-cache": "on", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_data", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/root/avocado/data/avocado-vt/data.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -object '{"qom-type": "iothread", "id": "iothread1"}' \
     -blockdev '{"node-name": "drive_data", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_data"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-blk-pci", "id": "data", "drive": "drive_data", "bootindex": 1, "write-cache": "on", "bus": "pcie-root-port-3", "addr": "0x0", "iothread": "iothread1"}' \
     -device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:51:97:3a:13:46", "id": "idH37UNL", "netdev": "idRruXsx", "bus": "pcie-root-port-4", "addr": "0x0"}'  \
     -netdev tap,id=idRruXsx,vhost=on  \
     -vnc :1  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 6}' \
     -incoming defer \
     -monitor stdio \

4.In src, create target backup node
  {"execute": "blockdev-create", "arguments": {"options": {"driver": "file", "filename": "/root/avocado/data/avocado-vt/base.qcow2", "size": 2147483648}, "job-id": "file_base"}, "id": "1er6syhm"}
  {"execute": "job-dismiss", "arguments": {"id": "file_base"}, "id": "8q5Eo5qm"}
  {"execute": "blockdev-add", "arguments": {"node-name": "file_base", "driver": "file", "filename": "/root/avocado/data/avocado-vt/base.qcow2", "aio": "threads", "auto-read-only": true, "discard": "unmap"}, "id": "RaYRu0EE"}
  {"execute": "blockdev-create", "arguments": {"options": {"driver": "qcow2", "file": "file_base", "size": 2147483648}, "job-id": "drive_base"}, "id": "zn5D27y0"}
  {"execute": "job-dismiss", "arguments": {"id": "drive_base"}, "id": "PWFf9V2s"}
  {"execute": "blockdev-add", "arguments": {"node-name": "drive_base", "driver": "qcow2", "file": "file_base", "read-only": false}, "id": "IlUFekXf"}

5.In src, do full backup and add a bitmap
  {"execute": "transaction", "arguments": {"actions": [{"type": "blockdev-backup", "data": {"device": "drive_data", "target": "drive_base", "job-id": "drive_data_1TrV", "sync": "full", "speed": 0, "compress": false, "auto-finalize": true, "auto-dismiss": true, "on-source-error": "report", "on-target-error": "report"}}, {"type": "block-dirty-bitmap-add", "data": {"node": "drive_data", "name": "bitmap_data"}}]}, "id": "hLS23EB5"}

6.After backup done, in src guest, create a new file
  (src guest)#dd if=/dev/urandom of=/mnt/vdb1/mlNx bs=1M count=10 oflag=direct
             #md5sum /mnt/vdb1/mlNx > /mnt/vdb1/mlNx.md5 && sync

7.In src, disable bitmap
  {"execute": "block-dirty-bitmap-disable", "arguments": {"node": "drive_data", "name": "bitmap_data"}, "id": "dXlh5Cm9"}

8.In src, set migraiton capabilities
  {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "pause-before-switchover"}]}, "id": "Ksu5d8sM"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "events"}]}, "id": "YqWcEVon"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "dirty-bitmaps"}]}, "id": "dG1Ier5x"}

9.In dst, set migraiton capabilities and set migrate-incoming
  {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "late-block-activate"}]}, "id": "67C8AC3j"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "events"}]}, "id": "YTZLJMTT"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "dirty-bitmaps"}]}, "id": "dzWpKBhh"}
{"execute": "migrate-incoming", "arguments": {"uri": "tcp:[::]:5531"}, "id": "Xu7H65iD"}

10.Migrate from src to dst.
  {"execute": "migrate", "arguments": {"uri": "tcp:localhost:5531", "blk": false, "inc": false}, "id": "4WC6sQfJ"}
{"timestamp": {"seconds": 1681445417, "microseconds": 781743}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}, "id": "4WC6sQfJ"}
{"timestamp": {"seconds": 1681445418, "microseconds": 12917}, "event": "MIGRATION_PASS", "data": {"pass": 1}}


Actual results:
After step10, qemu coredump
Src qmp monitor output:
 {"timestamp": {"seconds": 1681445417, "microseconds": 781743}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}, "id": "4WC6sQfJ"}
{"timestamp": {"seconds": 1681445418, "microseconds": 12917}, "event": "MIGRATION_PASS", "data": {"pass": 1}}

Dst qmp monitor output:
 {"timestamp": {"seconds": 1681445408, "microseconds": 164487}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}, "id": "Xu7H65iD"}
{"timestamp": {"seconds": 1681445417, "microseconds": 782170}, "event": "MIGRATION", "data": {"status": "active"}}
{"timestamp": {"seconds": 1681445418, "microseconds": 71210}, "event": "MIGRATION", "data": {"status": "failed"}}


Qemu coredump with info:
 qemu: qemu_mutex_unlock_impl: Operation not permitted
src.txt: line 49: 510966 Aborted                 (core dumped) /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' ...

Coredump info:
Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-44.scope
          Unit: session-44.scope
         Slice: user-0.slice
       Session: 44
     Owner UID: 0 (root)
       Boot ID: 3fc872d61fd14fc3ae2cdf1cf487f0f7
    Machine ID: 58bf48c028d34ee69040709abf09d8ec
      Hostname: dell-per440-08.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.3fc872d61fd14fc3ae2cdf1cf487f0f7.510966.1681376197000000.zst (present)
  Size on Disk: 654.9M
       Message: Process 510966 (qemu-kvm) of user 0 dumped core.
                
                Stack trace of thread 511408:
                #0  0x00007fb843ca157c __pthread_kill_implementation (libc.so.6 + 0xa157c)
                #1  0x00007fb843c54d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007fb843c287f3 abort (libc.so.6 + 0x287f3)
                #3  0x000055b9e1a8b112 qemu_mutex_unlock_impl (qemu-kvm + 0x9e8112)
                #4  0x000055b9e18962cd bdrv_poll_co (qemu-kvm + 0x7f32cd)
                #5  0x000055b9e1890e2e bdrv_nb_sectors (qemu-kvm + 0x7ede2e)
                #6  0x000055b9e15c50bb add_bitmaps_to_list (qemu-kvm + 0x5220bb)
                #7  0x000055b9e15c3a34 dirty_bitmap_save_setup (qemu-kvm + 0x520a34)
                #8  0x000055b9e15e35f2 qemu_savevm_state_setup (qemu-kvm + 0x5405f2)
                #9  0x000055b9e15d5053 migration_thread (qemu-kvm + 0x532053)
                #10 0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #11 0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #12 0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510969:
                #0  0x00007fb843d42a6e ppoll (libc.so.6 + 0x142a6e)
                #1  0x000055b9e1a88bee fdmon_poll_wait (qemu-kvm + 0x9e5bee)
                #2  0x000055b9e1a883c6 aio_poll (qemu-kvm + 0x9e53c6)
                #3  0x000055b9e188eee2 iothread_run (qemu-kvm + 0x7ebee2)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510966:
                #0  0x00007fb843c9c590 __GI___lll_lock_wait (libc.so.6 + 0x9c590)
                #1  0x00007fb843ca2c52 __pthread_mutex_lock.5 (libc.so.6 + 0xa2c52)
                #2  0x000055b9e1a8af6f qemu_mutex_lock_impl (qemu-kvm + 0x9e7f6f)
                #3  0x000055b9e1aa419d main_loop_wait (qemu-kvm + 0xa0119d)
                #4  0x000055b9e15a0c47 qemu_main_loop (qemu-kvm + 0x4fdc47)
                #5  0x000055b9e14169ea qemu_default_main (qemu-kvm + 0x3739ea)
                #6  0x00007fb843c3feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #7  0x00007fb843c3ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #8  0x000055b9e1416145 _start (qemu-kvm + 0x373145)
                
                Stack trace of thread 510981:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510983:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510978:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510975:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510967:
                #0  0x00007fb843d13945 clock_nanosleep.5 (libc.so.6 + 0x113945)
                #1  0x00007fb843d184f7 __nanosleep (libc.so.6 + 0x1184f7)
                #2  0x00007fb844303b07 g_usleep (libglib-2.0.so.0 + 0x7eb07)
                #3  0x000055b9e1a981ca call_rcu_thread (qemu-kvm + 0x9f51ca)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510982:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510976:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510979:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510968:
                #0  0x00007fb843d42a6e ppoll (libc.so.6 + 0x142a6e)
                #1  0x000055b9e1a88bee fdmon_poll_wait (qemu-kvm + 0x9e5bee)
                #2  0x000055b9e1a883c6 aio_poll (qemu-kvm + 0x9e53c6)
                #3  0x000055b9e188eee2 iothread_run (qemu-kvm + 0x7ebee2)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511073:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510974:
                #0  0x00007fb843d4296f __poll (libc.so.6 + 0x14296f)
                #1  0x00007fb84432e49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c)
                #2  0x00007fb8442d9483 g_main_loop_run (libglib-2.0.so.0 + 0x54483)
                #3  0x000055b9e188eeff iothread_run (qemu-kvm + 0x7ebeff)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510977:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510986:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9ebd0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9ebd0)
                #2  0x000055b9e1a8b4bf qemu_cond_wait_impl (qemu-kvm + 0x9e84bf)
                #3  0x000055b9e1450116 vnc_worker_thread (qemu-kvm + 0x3ad116)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511077:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511083:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510984:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511146:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511054:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511086:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511144:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511087:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511081:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511082:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511145:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511075:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511154:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511076:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511122:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511151:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511085:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511080:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511124:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 511079:
                #0  0x00007fb843c9c3ca __futex_abstimed_wait_common (libc.so.6 + 0x9c3ca)
                #1  0x00007fb843c9eed4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eed4)
                #2  0x000055b9e1a8b65c qemu_cond_timedwait_ts (qemu-kvm + 0x9e865c)
                #3  0x000055b9e1a8b600 qemu_cond_timedwait_impl (qemu-kvm + 0x9e8600)
                #4  0x000055b9e1aa6bd7 worker_thread (qemu-kvm + 0xa03bd7)
                #5  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #6  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 510980:
                #0  0x00007fb843c3ec6b ioctl (libc.so.6 + 0x3ec6b)
                #1  0x000055b9e182d98b kvm_vcpu_ioctl (qemu-kvm + 0x78a98b)
                #2  0x000055b9e1833191 kvm_cpu_exec (qemu-kvm + 0x790191)
                #3  0x000055b9e18357aa kvm_vcpu_thread_fn (qemu-kvm + 0x7927aa)
                #4  0x000055b9e1a8bf0a qemu_thread_start (qemu-kvm + 0x9e8f0a)
                #5  0x00007fb843c9f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007fb843c3f450 __clone3 (libc.so.6 + 0x3f450)
                ELF object binary architecture: AMD x86-64

Expected results:
Migration can be executed successfully.

Additional info:
 Will provide coredum file later.

Comment 2 aihua liang 2023-04-14 07:27:27 UTC
Not hit this issue on qemu-kvm-7.2.0-14.el9_2 after 20 times test by auto:qemu-kvm-7.2.0-14.el9_2, so it's a regression issue.

Comment 3 aihua liang 2023-04-14 07:49:18 UTC
Both virtio_blk and virtio_scsi hit this issue.

Comment 5 aihua liang 2023-04-14 10:35:09 UTC
Correct the auto script to:blockdev_inc_backup_with_migration

Comment 9 aihua liang 2023-06-18 10:59:23 UTC
Run test case:blockdev_inc_backup_with_migration for 100 times on qemu-kvm-8.0.0-5.el9, all pass.
 (099/100) repeat99.Host_RHEL.m9.u3.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.RHEL.9.3.0.x86_64.io-github-autotest-qemu.blockdev_inc_backup_with_migration.q35: PASS (182.33 s)
 (100/100) repeat100.Host_RHEL.m9.u3.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.RHEL.9.3.0.x86_64.io-github-autotest-qemu.blockdev_inc_backup_with_migration.q35: PASS (185.40 s)
RESULTS    : PASS 100 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /root/avocado/job-results/job-2023-06-18T01.02-ac2e91b/results.html
JOB TIME   : 19388.16 s

And run regression test, all pass.

So will close this bug as currentrelease.


Note You need to log in before you can comment on or make changes to this bug.