RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2090998 - [qemu-kvm] rhel guest failed boot with multi disks on error Failed to start udev Wait for Complete Device Initialization
Summary: [qemu-kvm] rhel guest failed boot with multi disks on error Failed to start u...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.1
Hardware: x86_64
OS: All
medium
high
Target Milestone: rc
: ---
Assignee: Stefan Hajnoczi
QA Contact: qing.wang
URL:
Whiteboard:
Depends On:
Blocks: 2211923
TreeView+ depends on / blocked
 
Reported: 2022-05-27 09:33 UTC by qing.wang
Modified: 2023-11-07 09:19 UTC (History)
16 users (show)

Fixed In Version: qemu-kvm-8.0.0-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2211923 (view as bug list)
Environment:
Last Closed: 2023-11-07 08:26:38 UTC
Type: ---
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
GDB Python script to check virtio-scsi-pci devices (2.58 KB, text/plain)
2023-03-22 14:05 UTC, Stefan Hajnoczi 		no flags Details
GDB Python script to print AioHandler in an IOThread (361 bytes, text/plain)
2023-03-22 14:06 UTC, Stefan Hajnoczi 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Gitlab redhat/rhel/src/qemu-kvm qemu-kvm merge_requests 269 0 None None None 2023-03-29 18:29:00 UTC
Red Hat Issue Tracker RHELPLAN-123544 0 None None None 2022-05-27 10:02:57 UTC
Red Hat Product Errata RHSA-2023:6368 0 None None None 2023-11-07 08:27:42 UTC

Description qing.wang 2022-05-27 09:33:47 UTC 
Description of problem:
Boot guest with multi disk and bus on AMD cpu host.
It failed on booting and kernel crash

2022-05-27 04:17:13: [K[    [0;31m*[0;1;31m*[0m] (2 of 2) A start job is running fordroot.device (1min 56s / no limit)
2022-05-27 04:17:13: [K[     [0;31m*[0m] (2 of 2) A start job is running fordroot.device (1min 57s / no limit)
2022-05-27 04:17:14: [K[    [0;31m*[0;1;31m*[0m] (1 of 2) A start job is running fore Initialization (1min 57s / 3min)
2022-05-27 04:17:14: [K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] (1 of 2) A start job is running fore Initialization (1min 58s / 3min)
2022-05-27 04:17:15: [K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] (1 of 2) A start job is running fore Initialization (1min 58s / 3min)
2022-05-27 04:17:15: [K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] (2 of 2) A start job is running fordroot.device (1min 59s / no limit)
2022-05-27 04:17:16: [K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] (2 of 2) A start job is running fordroot.device (1min 59s / no limit)
2022-05-27 04:17:16: [K[[0;1;31m*[0m[0;31m*    [0m] (2 of 2) A start job is running for5\x2droot.device (2min / no limit)
2022-05-27 04:17:17: [K[[0m[0;31m*     [0m] (1 of 2) A start job is running forevice Initialization (2min / 3min)[K[[0;1;31mFAILED[0m] Failed to start udev Wait for Complete Device Initialization.
2022-05-27 04:17:17: See 'systemctl status systemd-udev-settle.service' for details.
2022-05-27 04:17:17:          Starting Device-Mapper Multipath Device Controller...
2022-05-27 04:17:17: [[0;32m  OK  [0m] Started Device-Mapper Multipath Device Controller.
2022-05-27 04:17:17: [[0;32m  OK  [0m] Reached target Local File Systems (Pre).
2022-05-27 04:17:17: [[0;32m  OK  [0m] Reached target Local File Systems.
2022-05-27 04:17:17:          Starting Create Volatile Files and Directories...
2022-05-27 04:17:17:          Starting Open-iSCSI...
2022-05-27 04:17:17: [[0;32m  OK  [0m] Started Open-iSCSI.
2022-05-27 04:17:17: [[0;32m  OK  [0m] Started Create Volatile Files and Directories.
2022-05-27 04:17:17: [[0;32m  OK  [0m] Reached target System Initialization.
2022-05-27 04:17:17: [[0;32m  OK  [0m] Reached target Basic System.
2022-05-27 04:17:17:          Starting dracut initqueue hook...
2022-05-27 04:17:17: [  123.304632] TECH PREVIEW: NVMe/TCP may not be fully supported.
2022-05-27 04:17:17: [  123.304632] Please review provided documentation for limitations.
2022-05-27 04:19:20: [  246.751028] INFO: task kworker/u32:1:108 blocked for more than 120 seconds.
2022-05-27 04:19:20: [  246.752331]       Tainted: G               X --------- -  - 4.18.0-372.9.1.el8.x86_64 #1
2022-05-27 04:19:20: [  246.753538] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2022-05-27 04:19:20: [  246.754708] task:kworker/u32:1   state:D stack:    0 pid:  108 ppid:     2 flags:0x80004000
2022-05-27 04:19:20: [  246.755941] Workqueue: events_unbound async_run_entry_fn
2022-05-27 04:19:20: [  246.756902] Call Trace:
2022-05-27 04:19:20: [  246.757604]  __schedule+0x2d1/0x830
2022-05-27 04:19:20: [  246.758407]  ? kfree+0x1b0/0x1f0
2022-05-27 04:19:20: [  246.759179]  schedule+0x35/0xa0
2022-05-27 04:19:20: [  246.759927]  schedule_timeout+0x274/0x300
2022-05-27 04:19:20: [  246.760754]  wait_for_completion+0x96/0x100
2022-05-27 04:19:20: [  246.761604]  do_scan_async+0x48/0x150
2022-05-27 04:19:20: [  246.762391]  async_run_entry_fn+0x39/0x160
2022-05-27 04:19:20: [  246.763208]  process_one_work+0x1a7/0x360
2022-05-27 04:19:20: [  246.764017]  worker_thread+0x30/0x390
2022-05-27 04:19:20: [  246.764781]  ? create_worker+0x1a0/0x1a0
2022-05-27 04:19:20: [  246.765575]  kthread+0x10a/0x120
2022-05-27 04:19:20: [  246.766308]  ? set_kthread_struct+0x40/0x40
2022-05-27 04:19:20: [  246.767128]  ret_from_fork+0x35/0x40
2022-05-27 04:19:20: [  246.767885] INFO: task kworker/u32:2:109 blocked for more than 120 seconds.
2022-05-27 04:19:20: [  246.768950]       Tainted: G               X --------- -  - 4.18.0-372.9.1.el8.x86_64 #1
2022-05-27 04:19:20: [  246.770120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2022-05-27 04:19:20: [  246.771273] task:kworker/u32:2   state:D stack:    0 pid:  109 ppid:     2 flags:0x80004000
2022-05-27 04:19:20: [  246.772487] Workqueue: events_unbound async_run_entry_fn
2022-05-27 04:19:20: [  246.773433] Call Trace:
2022-05-27 04:19:20: [  246.774127]  __schedule+0x2d1/0x830
2022-05-27 04:19:20: [  246.774908]  ? kfree+0x1b0/0x1f0
2022-05-27 04:19:20: [  246.775666]  schedule+0x35/0xa0
2022-05-27 04:19:20: [  246.776420]  schedule_timeout+0x274/0x300
2022-05-27 04:19:20: [  246.777259]  ? __switch_to_asm+0x41/0x70
2022-05-27 04:19:20: [  246.778084]  ? reschedule_interrupt+0xa/0x20
2022-05-27 04:19:20: [  246.778926]  wait_for_completion+0x96/0x100
2022-05-27 04:19:20: [  246.779764]  do_scan_async+0x48/0x150
2022-05-27 04:19:20: [  246.780550]  async_run_entry_fn+0x39/0x160
2022-05-27 04:19:20: [  246.781368]  process_one_work+0x1a7/0x360
2022-05-27 04:19:20: [  246.782171]  worker_thread+0x30/0x390
2022-05-27 04:19:20: [  246.782926]  ? create_worker+0x1a0/0x1a0
2022-05-27 04:19:20: [  246.783709]  kthread+0x10a/0x120
2022-05-27 04:19:20: [  246.784428]  ? set_kthread_struct+0x40/0x40
2022-05-27 04:19:20: [  246.785229]  ret_from_fork+0x35/0x40
2022-05-27 04:19:20: [  246.785970] INFO: task kworker/u32:3:110 blocked for more than 120 seconds.
2022-05-27 04:19:20: [  246.787022]       Tainted: G               X --------


Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux release 9.1 Beta (Plow)
5.14.0-96.el9.x86_64
qemu-kvm-7.0.0-4.el9.x86_64
seabios-bin-1.16.0-2.el9.noarch
edk2-ovmf-20220221gitb24306f15d-1.el9.noarch
virtio-win-prewhql-0.1-219.iso

Guest:rhel8.6 or rhel9.1

How reproducible:
100%

Steps to Reproduce:
1.boot vm 

	[ -f /home/kvm_autotest_root/images/OVMF_VARS_x.fd ] || {  cp -f /usr/share/OVMF/OVMF_VARS.fd /home/kvm_autotest_root/images/OVMF_VARS_x.fd; }

 /usr/libexec/qemu-kvm \
 -name 'avocado-vt-vm1'  \
 -sandbox on  \
 -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
 -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
 -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kvm_autotest_root/images/OVMF_VARS_x.fd,auto-read-only=on,discard=unmap \
 -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
 -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
 -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
 -nodefaults \
 -device VGA,bus=pcie.0,addr=0x2 \
 -m 12288 \
 -object memory-backend-ram,size=12288M,id=mem-machine_mem  \
 -smp 16 \
 -cpu 'EPYC-Rome',+kvm_pv_unhalt \
 \
 -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
 -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
 -object iothread,id=iothread0 \
 -object iothread,id=iothread1 \
 -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
 -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/rhel910-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
 -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,write-cache=on \
 -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
 -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-3,addr=0x0,iothread=iothread1 \
 -blockdev node-name=file_stg0,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg0.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg0,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg0 \
 -device scsi-hd,scsi-id=0,lun=0,id=stg0,bus=virtio_scsi_pci1.0,drive=drive_stg0,write-cache=on \
 -blockdev node-name=file_stg1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg1.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg1 \
 -device scsi-hd,scsi-id=0,lun=8191,id=stg1,bus=virtio_scsi_pci1.0,drive=drive_stg1,write-cache=on \
 -blockdev node-name=file_stg2,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg2.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg2,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg2 \
 -device scsi-hd,scsi-id=0,lun=16382,id=stg2,bus=virtio_scsi_pci1.0,drive=drive_stg2,write-cache=on \
 -blockdev node-name=file_stg3,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg3.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg3,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg3 \
 -device scsi-hd,scsi-id=127,lun=0,id=stg3,bus=virtio_scsi_pci1.0,drive=drive_stg3,write-cache=on \
 -blockdev node-name=file_stg4,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg4.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg4,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg4 \
 -device scsi-hd,scsi-id=127,lun=8191,id=stg4,bus=virtio_scsi_pci1.0,drive=drive_stg4,write-cache=on \
 -blockdev node-name=file_stg5,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg5.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg5,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg5 \
 -device scsi-hd,scsi-id=127,lun=16382,id=stg5,bus=virtio_scsi_pci1.0,drive=drive_stg5,write-cache=on \
 -blockdev node-name=file_stg6,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg6.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg6,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg6 \
 -device scsi-hd,scsi-id=254,lun=0,id=stg6,bus=virtio_scsi_pci1.0,drive=drive_stg6,write-cache=on \
 -blockdev node-name=file_stg7,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg7.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg7,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg7 \
 -device scsi-hd,scsi-id=254,lun=8191,id=stg7,bus=virtio_scsi_pci1.0,drive=drive_stg7,write-cache=on \
 -blockdev node-name=file_stg8,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg8.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg8,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg8 \
 -device scsi-hd,scsi-id=254,lun=16382,id=stg8,bus=virtio_scsi_pci1.0,drive=drive_stg8,write-cache=on \
 -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
 -device virtio-scsi-pci,id=virtio_scsi_pci2,bus=pcie-root-port-4,addr=0x0,iothread=iothread0 \
 -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \
 -device virtio-scsi-pci,id=virtio_scsi_pci3,bus=pcie-root-port-5,addr=0x0,iothread=iothread0 \
 -blockdev node-name=file_stg9,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg9.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg9,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg9 \
 -device scsi-hd,scsi-id=0,lun=0,id=stg9,bus=virtio_scsi_pci3.0,drive=drive_stg9,write-cache=on \
 -blockdev node-name=file_stg10,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg10.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg10,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg10 \
 -device scsi-hd,scsi-id=0,lun=8191,id=stg10,bus=virtio_scsi_pci3.0,drive=drive_stg10,write-cache=on \
 -blockdev node-name=file_stg11,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg11.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg11,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg11 \
 -device scsi-hd,scsi-id=0,lun=16382,id=stg11,bus=virtio_scsi_pci3.0,drive=drive_stg11,write-cache=on \
 -blockdev node-name=file_stg12,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg12.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg12,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg12 \
 -device scsi-hd,scsi-id=127,lun=0,id=stg12,bus=virtio_scsi_pci3.0,drive=drive_stg12,write-cache=on \
 -blockdev node-name=file_stg13,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg13.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg13,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg13 \
 -device scsi-hd,scsi-id=127,lun=8191,id=stg13,bus=virtio_scsi_pci3.0,drive=drive_stg13,write-cache=on \
 -blockdev node-name=file_stg14,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg14.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg14,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg14 \
 -device scsi-hd,scsi-id=127,lun=16382,id=stg14,bus=virtio_scsi_pci3.0,drive=drive_stg14,write-cache=on \
 -blockdev node-name=file_stg15,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg15.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg15,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg15 \
 -device scsi-hd,scsi-id=254,lun=0,id=stg15,bus=virtio_scsi_pci3.0,drive=drive_stg15,write-cache=on \
 -blockdev node-name=file_stg16,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg16.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg16,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg16 \
 -device scsi-hd,scsi-id=254,lun=8191,id=stg16,bus=virtio_scsi_pci3.0,drive=drive_stg16,write-cache=on \
 -blockdev node-name=file_stg17,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg17.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg17,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg17 \
 -device scsi-hd,scsi-id=254,lun=16382,id=stg17,bus=virtio_scsi_pci3.0,drive=drive_stg17,write-cache=on \
 -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x1.0x6,bus=pcie.0,chassis=7 \
 -device virtio-scsi-pci,id=virtio_scsi_pci4,bus=pcie-root-port-6,addr=0x0,iothread=iothread1 \
 -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x1.0x7,bus=pcie.0,chassis=8 \
 -device virtio-scsi-pci,id=virtio_scsi_pci5,bus=pcie-root-port-7,addr=0x0,iothread=iothread1 \
 -blockdev node-name=file_stg18,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg18.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg18,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg18 \
 -device scsi-hd,scsi-id=0,lun=0,id=stg18,bus=virtio_scsi_pci5.0,drive=drive_stg18,write-cache=on \
 -blockdev node-name=file_stg19,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg19.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg19,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg19 \
 -device scsi-hd,scsi-id=0,lun=8191,id=stg19,bus=virtio_scsi_pci5.0,drive=drive_stg19,write-cache=on \
 -blockdev node-name=file_stg20,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg20.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg20,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg20 \
 -device scsi-hd,scsi-id=0,lun=16382,id=stg20,bus=virtio_scsi_pci5.0,drive=drive_stg20,write-cache=on \
 -blockdev node-name=file_stg21,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg21.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg21,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg21 \
 -device scsi-hd,scsi-id=127,lun=0,id=stg21,bus=virtio_scsi_pci5.0,drive=drive_stg21,write-cache=on \
 -blockdev node-name=file_stg22,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg22.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg22,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg22 \
 -device scsi-hd,scsi-id=127,lun=8191,id=stg22,bus=virtio_scsi_pci5.0,drive=drive_stg22,write-cache=on \
 -blockdev node-name=file_stg23,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg23.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg23,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg23 \
 -device scsi-hd,scsi-id=127,lun=16382,id=stg23,bus=virtio_scsi_pci5.0,drive=drive_stg23,write-cache=on \
 -blockdev node-name=file_stg24,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg24.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg24,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg24 \
 -device scsi-hd,scsi-id=254,lun=0,id=stg24,bus=virtio_scsi_pci5.0,drive=drive_stg24,write-cache=on \
 -blockdev node-name=file_stg25,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg25.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg25,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg25 \
 -device scsi-hd,scsi-id=254,lun=8191,id=stg25,bus=virtio_scsi_pci5.0,drive=drive_stg25,write-cache=on \
 -blockdev node-name=file_stg26,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg26.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg26,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg26 \
 -device scsi-hd,scsi-id=254,lun=16382,id=stg26,bus=virtio_scsi_pci5.0,drive=drive_stg26,write-cache=on \
 -device pcie-root-port,id=pcie-root-port-8,port=0x8,multifunction=on,bus=pcie.0,addr=0x3,chassis=9 \
 -device virtio-scsi-pci,id=virtio_scsi_pci6,bus=pcie-root-port-8,addr=0x0,iothread=iothread0 \
 -device pcie-root-port,id=pcie-root-port-9,port=0x9,addr=0x3.0x1,bus=pcie.0,chassis=10 \
 -device virtio-scsi-pci,id=virtio_scsi_pci7,bus=pcie-root-port-9,addr=0x0,iothread=iothread0 \
 -blockdev node-name=file_stg27,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg27.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg27,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg27 \
 -device scsi-hd,scsi-id=0,lun=0,id=stg27,bus=virtio_scsi_pci7.0,drive=drive_stg27,write-cache=on \
 -blockdev node-name=file_stg28,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg28.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg28,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg28 \
 -device scsi-hd,scsi-id=0,lun=8191,id=stg28,bus=virtio_scsi_pci7.0,drive=drive_stg28,write-cache=on \
 -blockdev node-name=file_stg29,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg29.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg29,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg29 \
 -device scsi-hd,scsi-id=0,lun=16382,id=stg29,bus=virtio_scsi_pci7.0,drive=drive_stg29,write-cache=on \
 -blockdev node-name=file_stg30,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg30.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg30,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg30 \
 -device scsi-hd,scsi-id=127,lun=0,id=stg30,bus=virtio_scsi_pci7.0,drive=drive_stg30,write-cache=on \
 -blockdev node-name=file_stg31,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg31.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg31,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg31 \
 -device scsi-hd,scsi-id=127,lun=8191,id=stg31,bus=virtio_scsi_pci7.0,drive=drive_stg31,write-cache=on \
 -blockdev node-name=file_stg32,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg32.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg32,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg32 \
 -device scsi-hd,scsi-id=127,lun=16382,id=stg32,bus=virtio_scsi_pci7.0,drive=drive_stg32,write-cache=on \
 -blockdev node-name=file_stg33,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg33.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg33,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg33 \
 -device scsi-hd,scsi-id=254,lun=0,id=stg33,bus=virtio_scsi_pci7.0,drive=drive_stg33,write-cache=on \
 -blockdev node-name=file_stg34,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg34.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg34,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg34 \
 -device scsi-hd,scsi-id=254,lun=8191,id=stg34,bus=virtio_scsi_pci7.0,drive=drive_stg34,write-cache=on \
 -blockdev node-name=file_stg35,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg35.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg35,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg35 \
 -device scsi-hd,scsi-id=254,lun=16382,id=stg35,bus=virtio_scsi_pci7.0,drive=drive_stg35,write-cache=on \
 -device pcie-root-port,id=pcie-root-port-10,port=0xa,addr=0x3.0x2,bus=pcie.0,chassis=11 \
 -device virtio-scsi-pci,id=virtio_scsi_pci8,bus=pcie-root-port-10,addr=0x0,iothread=iothread1 \
 -device pcie-root-port,id=pcie-root-port-11,port=0xb,addr=0x3.0x3,bus=pcie.0,chassis=12 \
 -device virtio-scsi-pci,id=virtio_scsi_pci9,bus=pcie-root-port-11,addr=0x0,iothread=iothread1 \
 -blockdev node-name=file_stg36,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg36.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg36,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg36 \
 -device scsi-hd,scsi-id=0,lun=0,id=stg36,bus=virtio_scsi_pci9.0,drive=drive_stg36,write-cache=on \
 -blockdev node-name=file_stg37,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg37.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg37,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg37 \
 -device scsi-hd,scsi-id=0,lun=8191,id=stg37,bus=virtio_scsi_pci9.0,drive=drive_stg37,write-cache=on \
 -blockdev node-name=file_stg38,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg38.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg38,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg38 \
 -device scsi-hd,scsi-id=0,lun=16382,id=stg38,bus=virtio_scsi_pci9.0,drive=drive_stg38,write-cache=on \
 -blockdev node-name=file_stg39,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg39.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg39,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg39 \
 -device scsi-hd,scsi-id=127,lun=0,id=stg39,bus=virtio_scsi_pci9.0,drive=drive_stg39,write-cache=on \
 -blockdev node-name=file_stg40,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg40.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg40,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg40 \
 -device scsi-hd,scsi-id=127,lun=8191,id=stg40,bus=virtio_scsi_pci9.0,drive=drive_stg40,write-cache=on \
 -blockdev node-name=file_stg41,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg41.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg41,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg41 \
 -device scsi-hd,scsi-id=127,lun=16382,id=stg41,bus=virtio_scsi_pci9.0,drive=drive_stg41,write-cache=on \
 -blockdev node-name=file_stg42,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg42.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg42,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg42 \
 -device scsi-hd,scsi-id=254,lun=0,id=stg42,bus=virtio_scsi_pci9.0,drive=drive_stg42,write-cache=on \
 -blockdev node-name=file_stg43,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg43.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg43,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg43 \
 -device scsi-hd,scsi-id=254,lun=8191,id=stg43,bus=virtio_scsi_pci9.0,drive=drive_stg43,write-cache=on \
 -blockdev node-name=file_stg44,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg44.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg44,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg44 \
 -device scsi-hd,scsi-id=254,lun=16382,id=stg44,bus=virtio_scsi_pci9.0,drive=drive_stg44,write-cache=on \
 -device pcie-root-port,id=pcie-root-port-12,port=0xc,addr=0x3.0x4,bus=pcie.0,chassis=13 \
 -device virtio-scsi-pci,id=virtio_scsi_pci10,bus=pcie-root-port-12,addr=0x0,iothread=iothread0 \
 -device pcie-root-port,id=pcie-root-port-13,port=0xd,addr=0x3.0x5,bus=pcie.0,chassis=14 \
 -device virtio-scsi-pci,id=virtio_scsi_pci11,bus=pcie-root-port-13,addr=0x0,iothread=iothread0 \
 -blockdev node-name=file_stg45,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg45.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg45,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg45 \
 -device scsi-hd,scsi-id=0,lun=0,id=stg45,bus=virtio_scsi_pci11.0,drive=drive_stg45,write-cache=on \
 -blockdev node-name=file_stg46,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg46.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg46,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg46 \
 -device scsi-hd,scsi-id=0,lun=8191,id=stg46,bus=virtio_scsi_pci11.0,drive=drive_stg46,write-cache=on \
 -blockdev node-name=file_stg47,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg47.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg47,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg47 \
 -device scsi-hd,scsi-id=0,lun=16382,id=stg47,bus=virtio_scsi_pci11.0,drive=drive_stg47,write-cache=on \
 -blockdev node-name=file_stg48,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg48.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg48,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg48 \
 -device scsi-hd,scsi-id=127,lun=0,id=stg48,bus=virtio_scsi_pci11.0,drive=drive_stg48,write-cache=on \
 -blockdev node-name=file_stg49,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg49.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg49,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg49 \
 -device scsi-hd,scsi-id=127,lun=8191,id=stg49,bus=virtio_scsi_pci11.0,drive=drive_stg49,write-cache=on \
 -blockdev node-name=file_stg50,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg50.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg50,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg50 \
 -device scsi-hd,scsi-id=127,lun=16382,id=stg50,bus=virtio_scsi_pci11.0,drive=drive_stg50,write-cache=on \
 -blockdev node-name=file_stg51,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg51.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg51,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg51 \
 -device scsi-hd,scsi-id=254,lun=0,id=stg51,bus=virtio_scsi_pci11.0,drive=drive_stg51,write-cache=on \
 -blockdev node-name=file_stg52,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg52.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg52,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg52 \
 -device scsi-hd,scsi-id=254,lun=8191,id=stg52,bus=virtio_scsi_pci11.0,drive=drive_stg52,write-cache=on \
 -blockdev node-name=file_stg53,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg53.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg53,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg53 \
 -device scsi-hd,scsi-id=254,lun=16382,id=stg53,bus=virtio_scsi_pci11.0,drive=drive_stg53,write-cache=on \
 -device pcie-root-port,id=pcie-root-port-14,port=0xe,addr=0x3.0x6,bus=pcie.0,chassis=15 \
 -device virtio-scsi-pci,id=virtio_scsi_pci12,bus=pcie-root-port-14,addr=0x0,iothread=iothread1 \
 -device pcie-root-port,id=pcie-root-port-15,port=0xf,addr=0x3.0x7,bus=pcie.0,chassis=16 \
 -device virtio-scsi-pci,id=virtio_scsi_pci13,bus=pcie-root-port-15,addr=0x0,iothread=iothread1 \
 -blockdev node-name=file_stg54,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg54.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg54,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg54 \
 -device scsi-hd,scsi-id=0,lun=0,id=stg54,bus=virtio_scsi_pci13.0,drive=drive_stg54,write-cache=on \
 -blockdev node-name=file_stg55,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg55.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg55,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg55 \
 -device scsi-hd,scsi-id=0,lun=8191,id=stg55,bus=virtio_scsi_pci13.0,drive=drive_stg55,write-cache=on \
 -blockdev node-name=file_stg56,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg56.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg56,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg56 \
 -device scsi-hd,scsi-id=0,lun=16382,id=stg56,bus=virtio_scsi_pci13.0,drive=drive_stg56,write-cache=on \
 -blockdev node-name=file_stg57,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg57.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg57,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg57 \
 -device scsi-hd,scsi-id=127,lun=0,id=stg57,bus=virtio_scsi_pci13.0,drive=drive_stg57,write-cache=on \
 -blockdev node-name=file_stg58,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg58.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg58,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg58 \
 -device scsi-hd,scsi-id=127,lun=8191,id=stg58,bus=virtio_scsi_pci13.0,drive=drive_stg58,write-cache=on \
 -blockdev node-name=file_stg59,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg59.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg59,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg59 \
 -device scsi-hd,scsi-id=127,lun=16382,id=stg59,bus=virtio_scsi_pci13.0,drive=drive_stg59,write-cache=on \
 -blockdev node-name=file_stg60,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg60.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg60,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg60 \
 -device scsi-hd,scsi-id=254,lun=0,id=stg60,bus=virtio_scsi_pci13.0,drive=drive_stg60,write-cache=on \
 -blockdev node-name=file_stg61,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg61.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg61,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg61 \
 -device scsi-hd,scsi-id=254,lun=8191,id=stg61,bus=virtio_scsi_pci13.0,drive=drive_stg61,write-cache=on \
 -blockdev node-name=file_stg62,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg62.qcow2,cache.direct=on,cache.no-flush=off \
 -blockdev node-name=drive_stg62,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg62 \
 -device scsi-hd,scsi-id=254,lun=16382,id=stg62,bus=virtio_scsi_pci13.0,drive=drive_stg62,write-cache=on \
 -device pcie-root-port,id=pcie-root-port-16,port=0x10,multifunction=on,bus=pcie.0,addr=0x4,chassis=17 \
 -device virtio-net-pci,mac=9a:33:ae:2d:a9:17,id=id6Gkz3l,netdev=idK0Huuo,bus=pcie-root-port-16,addr=0x0  \
 -netdev tap,id=idK0Huuo,vhost=on   \
 -vnc :5  \
 -qmp tcp:0:5955,server=on,wait=off  \
 -monitor stdio \
 -rtc base=utc,clock=host,driftfix=slew  \
 -boot menu=off,order=cdn,once=c,strict=off \
 -enable-kvm \
 -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x5,chassis=18


2.
3.

Actual results:
boot hang and kernel crash

Expected results:
boot succeed

Additional info:
cpu info
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         43 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  32
  On-line CPU(s) list:   0-31
Vendor ID:               AuthenticAMD
  BIOS Vendor ID:        AMD
  Model name:            AMD EPYC 7302P 16-Core Processor
    BIOS Model name:     AMD EPYC 7302P 16-Core Processor               
    CPU family:          23
    Model:               49
    Thread(s) per core:  2
    Core(s) per socket:  16
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            5989.05
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr ss
                         e sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_t
                         sc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2a
                         pic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm ss
                         e4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext
                          perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase b
                         mi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1
                          xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wb
                         noinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeass
                         ists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_reco
                         v succor smca sme sev sev_es


1.boot with seabios no issue found
2.if remove smp or change smp to 2 , it may pass
3.Same test pass on other qemu/ovmf but the cpu is not same  (intel)

Red Hat Enterprise Linux release 9.0 (Plow)
5.14.0-70.13.1.el9_0.x86_64
qemu-kvm-6.2.0-11.el9_0.2.x86_64
seabios-bin-1.15.0-1.el9.noarch
edk2-ovmf-20220126gitbb1bba3d77-3.el9.noarch
virtio-win-prewhql-0.1-219.iso


Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  20
  On-line CPU(s) list:   0-19
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel
  Model name:            Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
    BIOS Model name:     Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
    CPU family:          6
    Model:               85
    Thread(s) per core:  2
    Core(s) per socket:  10
    Socket(s):           1
    Stepping:            7
    CPU max MHz:         3200.0000
    CPU min MHz:         1000.0000
    BogoMIPS:            4400.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mm
                         x fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon peb
                         s bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_
                         cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcn
                         t tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ca
                         t_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnm
                         i flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx
                          rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl
                          xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm id
                         a arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Comment 3 qing.wang 2022-05-27 09:59:55 UTC 
Passed on other AMD host,

Red Hat Enterprise Linux release 8.6 (Ootpa)
4.18.0-372.10.1.el8_6.x86_64
qemu-kvm-6.2.0-11.module+el8.6.0+14712+f96656d3.x86_64
seabios-bin-1.15.0-2.module+el8.6.0+14757+c25ee005.noarch
edk2-ovmf-20220126gitbb1bba3d77-2.el8.noarch
python3-libvirt-8.0.0-1.module+el8.6.0+13896+a8fa8f67.x86_64
virtio-win-prewhql-0.1-219.iso


Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              32
On-line CPU(s) list: 0-31
Thread(s) per core:  2
Core(s) per socket:  16
Socket(s):           1
NUMA node(s):        1
Vendor ID:           AuthenticAMD
BIOS Vendor ID:      AMD
CPU family:          23
Model:               49
Model name:          AMD EPYC 7302P 16-Core Processor
BIOS Model name:     AMD EPYC 7302P 16-Core Processor               
Stepping:            0
CPU MHz:             2994.247
BogoMIPS:            5988.49
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-31


It looks like regression issue ?
Comment 4 Klaus Heinrich Kiwi 2022-06-10 12:37:52 UTC 
(In reply to qing.wang from comment #3)
> Passed on other AMD host,
> 
> Red Hat Enterprise Linux release 8.6 (Ootpa)
> 4.18.0-372.10.1.el8_6.x86_64
> qemu-kvm-6.2.0-11.module+el8.6.0+14712+f96656d3.x86_64
> seabios-bin-1.15.0-2.module+el8.6.0+14757+c25ee005.noarch
> edk2-ovmf-20220126gitbb1bba3d77-2.el8.noarch
> python3-libvirt-8.0.0-1.module+el8.6.0+13896+a8fa8f67.x86_64
> virtio-win-prewhql-0.1-219.iso
> 
> 
> Architecture:        x86_64
> CPU op-mode(s):      32-bit, 64-bit
> Byte Order:          Little Endian
> CPU(s):              32
> On-line CPU(s) list: 0-31
> Thread(s) per core:  2
> Core(s) per socket:  16
> Socket(s):           1
> NUMA node(s):        1
> Vendor ID:           AuthenticAMD
> BIOS Vendor ID:      AMD
> CPU family:          23
> Model:               49
> Model name:          AMD EPYC 7302P 16-Core Processor
> BIOS Model name:     AMD EPYC 7302P 16-Core Processor               
> Stepping:            0
> CPU MHz:             2994.247
> BogoMIPS:            5988.49
> Virtualization:      AMD-V
> L1d cache:           32K
> L1i cache:           32K
> L2 cache:            512K
> L3 cache:            16384K
> NUMA node0 CPU(s):   0-31
> 
> 
> It looks like regression issue ?

Can you qualify that further? Because it looks like the HOST cpu is relevant to triggering the issue, but I'm not sure we are seeing evidence of regression in the software (i.e., are you able to pass the test on a hardware that demonstrably fails, but using an older version for OVMF?)

Gerd, this is an oddball, can you take it?

 -Klaus
Comment 5 Gerd Hoffmann 2022-06-13 08:53:45 UTC 
> > It looks like regression issue ?
> 
> Can you qualify that further? Because it looks like the HOST cpu is relevant
> to triggering the issue, but I'm not sure we are seeing evidence of
> regression in the software (i.e., are you able to pass the test on a
> hardware that demonstrably fails, but using an older version for OVMF?)

Guest kernel apparently waiting for disk i/o.  Not obvious what the root cause is.
Given the same test works fine on another amd box it could be bad hardware.

So, yes, we need more info to qualify whenever this is a regression or not.
Does the test work (on the same hardware) reliable with older versions?
Can you try run 'memtest' on the machine where the test fails?
Comment 6 qing.wang 2022-06-13 10:04:24 UTC 
I run same testing on host as #0  with different qemu.(ovmf using same)

Not issue:
Red Hat Enterprise Linux release 9.0 (Plow)
5.14.0-70.15.1.el9_0.x86_64
qemu-kvm-6.2.0-11.el9_0.3.x86_64
seabios-bin-1.15.0-1.el9.noarch
edk2-ovmf-20220126gitbb1bba3d77-3.el9.noarch
virtio-win-prewhql-0.1-221.iso


Hit issue:
Red Hat Enterprise Linux release 9.0 (Plow)
5.14.0-70.15.1.el9_0.x86_64
qemu-kvm-7.0.0-5.el9.x86_64
seabios-bin-1.15.0-1.el9.noarch
edk2-ovmf-20220126gitbb1bba3d77-3.el9.noarch
virtio-win-prewhql-0.1-221.iso
Comment 7 Gerd Hoffmann 2022-06-13 10:45:44 UTC 
> I run same testing on host as #0  with different qemu.(ovmf using same)

> Not issue:
> qemu-kvm-6.2.0-11.el9_0.3.x86_64

> Hit issue:
> qemu-kvm-7.0.0-5.el9.x86_64

So, the qemu rebase apparently broke this.
Why is this tagged as ovmf issue?
Does the same configuration with seabios (and qemu 7.0) work fine?
Comment 8 qing.wang 2022-06-14 07:16:38 UTC 
(In reply to Gerd Hoffmann from comment #7)
> > I run same testing on host as #0  with different qemu.(ovmf using same)
> 
> > Not issue:
> > qemu-kvm-6.2.0-11.el9_0.3.x86_64
> 
> > Hit issue:
> > qemu-kvm-7.0.0-5.el9.x86_64
> 
> So, the qemu rebase apparently broke this.
> Why is this tagged as ovmf issue?
> Does the same configuration with seabios (and qemu 7.0) work fine?

This issue related to multi factors:iothread , aio ,guest,ovmf

scenario 1 : seabios +iothread +aio_native+rhel9, no issue

/usr/libexec/qemu-kvm \
     -name 'avocado-vt-bios'  \
     -sandbox on  \
     -machine q35,memory-backend=mem-machine_mem \
     -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
     -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
     -nodefaults \
     -device VGA,bus=pcie.0,addr=0x2 \
     -m 12G \
     -object memory-backend-ram,size=12G,id=mem-machine_mem  \
     -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
     -cpu 'EPYC-Rome',+kvm_pv_unhalt \
     -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
     -object iothread,id=iothread0 \
     -object iothread,id=iothread1 \
     -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
     -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
     -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi-bios.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
     -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
     -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-3,addr=0x0,iothread=iothread1 \
     -blockdev node-name=file_stg0,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg0.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg0,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg0 \
     -device scsi-hd,scsi-id=0,lun=0,id=stg0,bus=virtio_scsi_pci1.0,drive=drive_stg0,write-cache=on \
     -blockdev node-name=file_stg1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg1.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg1 \
     -device scsi-hd,scsi-id=0,lun=8191,id=stg1,bus=virtio_scsi_pci1.0,drive=drive_stg1,write-cache=on \
     -blockdev node-name=file_stg2,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg2.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg2,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg2 \
     -device scsi-hd,scsi-id=0,lun=16382,id=stg2,bus=virtio_scsi_pci1.0,drive=drive_stg2,write-cache=on \
     -blockdev node-name=file_stg3,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg3.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg3,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg3 \
     -device scsi-hd,scsi-id=127,lun=0,id=stg3,bus=virtio_scsi_pci1.0,drive=drive_stg3,write-cache=on \
     -blockdev node-name=file_stg4,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg4.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg4,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg4 \
     -device scsi-hd,scsi-id=127,lun=8191,id=stg4,bus=virtio_scsi_pci1.0,drive=drive_stg4,write-cache=on \
     -blockdev node-name=file_stg5,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg5.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg5,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg5 \
     -device scsi-hd,scsi-id=127,lun=16382,id=stg5,bus=virtio_scsi_pci1.0,drive=drive_stg5,write-cache=on \
     -blockdev node-name=file_stg6,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg6.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg6,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg6 \
     -device scsi-hd,scsi-id=254,lun=0,id=stg6,bus=virtio_scsi_pci1.0,drive=drive_stg6,write-cache=on \
     -blockdev node-name=file_stg7,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg7.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg7,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg7 \
     -device scsi-hd,scsi-id=254,lun=8191,id=stg7,bus=virtio_scsi_pci1.0,drive=drive_stg7,write-cache=on \
     -blockdev node-name=file_stg8,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg8.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg8,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg8 \
     -device scsi-hd,scsi-id=254,lun=16382,id=stg8,bus=virtio_scsi_pci1.0,drive=drive_stg8,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
     -device virtio-scsi-pci,id=virtio_scsi_pci2,bus=pcie-root-port-4,addr=0x0,iothread=iothread0 \
     -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \
     -device virtio-scsi-pci,id=virtio_scsi_pci3,bus=pcie-root-port-5,addr=0x0,iothread=iothread0 \
     -blockdev node-name=file_stg9,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg9.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg9,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg9 \
     -device scsi-hd,scsi-id=0,lun=0,id=stg9,bus=virtio_scsi_pci3.0,drive=drive_stg9,write-cache=on \
     -blockdev node-name=file_stg10,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg10.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg10,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg10 \
     -device scsi-hd,scsi-id=0,lun=8191,id=stg10,bus=virtio_scsi_pci3.0,drive=drive_stg10,write-cache=on \
     -blockdev node-name=file_stg11,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg11.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg11,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg11 \
     -device scsi-hd,scsi-id=0,lun=16382,id=stg11,bus=virtio_scsi_pci3.0,drive=drive_stg11,write-cache=on \
     -blockdev node-name=file_stg12,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg12.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg12,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg12 \
     -device scsi-hd,scsi-id=127,lun=0,id=stg12,bus=virtio_scsi_pci3.0,drive=drive_stg12,write-cache=on \
     -blockdev node-name=file_stg13,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg13.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg13,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg13 \
     -device scsi-hd,scsi-id=127,lun=8191,id=stg13,bus=virtio_scsi_pci3.0,drive=drive_stg13,write-cache=on \
     -blockdev node-name=file_stg14,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg14.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg14,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg14 \
     -device scsi-hd,scsi-id=127,lun=16382,id=stg14,bus=virtio_scsi_pci3.0,drive=drive_stg14,write-cache=on \
     -blockdev node-name=file_stg15,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg15.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg15,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg15 \
     -device scsi-hd,scsi-id=254,lun=0,id=stg15,bus=virtio_scsi_pci3.0,drive=drive_stg15,write-cache=on \
     -blockdev node-name=file_stg16,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg16.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg16,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg16 \
     -device scsi-hd,scsi-id=254,lun=8191,id=stg16,bus=virtio_scsi_pci3.0,drive=drive_stg16,write-cache=on \
     -blockdev node-name=file_stg17,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg17.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg17,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg17 \
     -device scsi-hd,scsi-id=254,lun=16382,id=stg17,bus=virtio_scsi_pci3.0,drive=drive_stg17,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x1.0x6,bus=pcie.0,chassis=7 \
     -device virtio-scsi-pci,id=virtio_scsi_pci4,bus=pcie-root-port-6,addr=0x0,iothread=iothread1 \
     -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x1.0x7,bus=pcie.0,chassis=8 \
     -device virtio-scsi-pci,id=virtio_scsi_pci5,bus=pcie-root-port-7,addr=0x0,iothread=iothread1 \
     -blockdev node-name=file_stg18,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg18.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg18,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg18 \
     -device scsi-hd,scsi-id=0,lun=0,id=stg18,bus=virtio_scsi_pci5.0,drive=drive_stg18,write-cache=on \
     -blockdev node-name=file_stg19,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg19.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg19,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg19 \
     -device scsi-hd,scsi-id=0,lun=8191,id=stg19,bus=virtio_scsi_pci5.0,drive=drive_stg19,write-cache=on \
     -blockdev node-name=file_stg20,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg20.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg20,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg20 \
     -device scsi-hd,scsi-id=0,lun=16382,id=stg20,bus=virtio_scsi_pci5.0,drive=drive_stg20,write-cache=on \
     -blockdev node-name=file_stg21,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg21.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg21,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg21 \
     -device scsi-hd,scsi-id=127,lun=0,id=stg21,bus=virtio_scsi_pci5.0,drive=drive_stg21,write-cache=on \
     -blockdev node-name=file_stg22,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg22.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg22,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg22 \
     -device scsi-hd,scsi-id=127,lun=8191,id=stg22,bus=virtio_scsi_pci5.0,drive=drive_stg22,write-cache=on \
     -blockdev node-name=file_stg23,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg23.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg23,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg23 \
     -device scsi-hd,scsi-id=127,lun=16382,id=stg23,bus=virtio_scsi_pci5.0,drive=drive_stg23,write-cache=on \
     -blockdev node-name=file_stg24,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg24.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg24,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg24 \
     -device scsi-hd,scsi-id=254,lun=0,id=stg24,bus=virtio_scsi_pci5.0,drive=drive_stg24,write-cache=on \
     -blockdev node-name=file_stg25,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg25.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg25,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg25 \
     -device scsi-hd,scsi-id=254,lun=8191,id=stg25,bus=virtio_scsi_pci5.0,drive=drive_stg25,write-cache=on \
     -blockdev node-name=file_stg26,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg26.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg26,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg26 \
     -device scsi-hd,scsi-id=254,lun=16382,id=stg26,bus=virtio_scsi_pci5.0,drive=drive_stg26,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-8,port=0x8,multifunction=on,bus=pcie.0,addr=0x3,chassis=9 \
     -device virtio-scsi-pci,id=virtio_scsi_pci6,bus=pcie-root-port-8,addr=0x0,iothread=iothread0 \
     -device pcie-root-port,id=pcie-root-port-9,port=0x9,addr=0x3.0x1,bus=pcie.0,chassis=10 \
     -device virtio-scsi-pci,id=virtio_scsi_pci7,bus=pcie-root-port-9,addr=0x0,iothread=iothread0 \
     -blockdev node-name=file_stg27,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg27.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg27,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg27 \
     -device scsi-hd,scsi-id=0,lun=0,id=stg27,bus=virtio_scsi_pci7.0,drive=drive_stg27,write-cache=on \
     -blockdev node-name=file_stg28,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg28.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg28,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg28 \
     -device scsi-hd,scsi-id=0,lun=8191,id=stg28,bus=virtio_scsi_pci7.0,drive=drive_stg28,write-cache=on \
     -blockdev node-name=file_stg29,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg29.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg29,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg29 \
     -device scsi-hd,scsi-id=0,lun=16382,id=stg29,bus=virtio_scsi_pci7.0,drive=drive_stg29,write-cache=on \
     -blockdev node-name=file_stg30,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg30.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg30,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg30 \
     -device scsi-hd,scsi-id=127,lun=0,id=stg30,bus=virtio_scsi_pci7.0,drive=drive_stg30,write-cache=on \
     -blockdev node-name=file_stg31,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg31.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg31,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg31 \
     -device scsi-hd,scsi-id=127,lun=8191,id=stg31,bus=virtio_scsi_pci7.0,drive=drive_stg31,write-cache=on \
     -blockdev node-name=file_stg32,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg32.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg32,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg32 \
     -device scsi-hd,scsi-id=127,lun=16382,id=stg32,bus=virtio_scsi_pci7.0,drive=drive_stg32,write-cache=on \
     -blockdev node-name=file_stg33,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg33.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg33,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg33 \
     -device scsi-hd,scsi-id=254,lun=0,id=stg33,bus=virtio_scsi_pci7.0,drive=drive_stg33,write-cache=on \
     -blockdev node-name=file_stg34,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg34.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg34,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg34 \
     -device scsi-hd,scsi-id=254,lun=8191,id=stg34,bus=virtio_scsi_pci7.0,drive=drive_stg34,write-cache=on \
     -blockdev node-name=file_stg35,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg35.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg35,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg35 \
     -device scsi-hd,scsi-id=254,lun=16382,id=stg35,bus=virtio_scsi_pci7.0,drive=drive_stg35,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-10,port=0xa,addr=0x3.0x2,bus=pcie.0,chassis=11 \
     -device virtio-scsi-pci,id=virtio_scsi_pci8,bus=pcie-root-port-10,addr=0x0,iothread=iothread1 \
     -device pcie-root-port,id=pcie-root-port-11,port=0xb,addr=0x3.0x3,bus=pcie.0,chassis=12 \
     -device virtio-scsi-pci,id=virtio_scsi_pci9,bus=pcie-root-port-11,addr=0x0,iothread=iothread1 \
     -blockdev node-name=file_stg36,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg36.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg36,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg36 \
     -device scsi-hd,scsi-id=0,lun=0,id=stg36,bus=virtio_scsi_pci9.0,drive=drive_stg36,write-cache=on \
     -blockdev node-name=file_stg37,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg37.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg37,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg37 \
     -device scsi-hd,scsi-id=0,lun=8191,id=stg37,bus=virtio_scsi_pci9.0,drive=drive_stg37,write-cache=on \
     -blockdev node-name=file_stg38,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg38.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg38,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg38 \
     -device scsi-hd,scsi-id=0,lun=16382,id=stg38,bus=virtio_scsi_pci9.0,drive=drive_stg38,write-cache=on \
     -blockdev node-name=file_stg39,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg39.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg39,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg39 \
     -device scsi-hd,scsi-id=127,lun=0,id=stg39,bus=virtio_scsi_pci9.0,drive=drive_stg39,write-cache=on \
     -blockdev node-name=file_stg40,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg40.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg40,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg40 \
     -device scsi-hd,scsi-id=127,lun=8191,id=stg40,bus=virtio_scsi_pci9.0,drive=drive_stg40,write-cache=on \
     -blockdev node-name=file_stg41,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg41.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg41,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg41 \
     -device scsi-hd,scsi-id=127,lun=16382,id=stg41,bus=virtio_scsi_pci9.0,drive=drive_stg41,write-cache=on \
     -blockdev node-name=file_stg42,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg42.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg42,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg42 \
     -device scsi-hd,scsi-id=254,lun=0,id=stg42,bus=virtio_scsi_pci9.0,drive=drive_stg42,write-cache=on \
     -blockdev node-name=file_stg43,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg43.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg43,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg43 \
     -device scsi-hd,scsi-id=254,lun=8191,id=stg43,bus=virtio_scsi_pci9.0,drive=drive_stg43,write-cache=on \
     -blockdev node-name=file_stg44,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg44.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg44,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg44 \
     -device scsi-hd,scsi-id=254,lun=16382,id=stg44,bus=virtio_scsi_pci9.0,drive=drive_stg44,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-12,port=0xc,addr=0x3.0x4,bus=pcie.0,chassis=13 \
     -device virtio-scsi-pci,id=virtio_scsi_pci10,bus=pcie-root-port-12,addr=0x0,iothread=iothread0 \
     -device pcie-root-port,id=pcie-root-port-13,port=0xd,addr=0x3.0x5,bus=pcie.0,chassis=14 \
     -device virtio-scsi-pci,id=virtio_scsi_pci11,bus=pcie-root-port-13,addr=0x0,iothread=iothread0 \
     -blockdev node-name=file_stg45,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg45.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg45,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg45 \
     -device scsi-hd,scsi-id=0,lun=0,id=stg45,bus=virtio_scsi_pci11.0,drive=drive_stg45,write-cache=on \
     -blockdev node-name=file_stg46,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg46.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg46,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg46 \
     -device scsi-hd,scsi-id=0,lun=8191,id=stg46,bus=virtio_scsi_pci11.0,drive=drive_stg46,write-cache=on \
     -blockdev node-name=file_stg47,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg47.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg47,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg47 \
     -device scsi-hd,scsi-id=0,lun=16382,id=stg47,bus=virtio_scsi_pci11.0,drive=drive_stg47,write-cache=on \
     -blockdev node-name=file_stg48,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg48.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg48,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg48 \
     -device scsi-hd,scsi-id=127,lun=0,id=stg48,bus=virtio_scsi_pci11.0,drive=drive_stg48,write-cache=on \
     -blockdev node-name=file_stg49,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg49.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg49,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg49 \
     -device scsi-hd,scsi-id=127,lun=8191,id=stg49,bus=virtio_scsi_pci11.0,drive=drive_stg49,write-cache=on \
     -blockdev node-name=file_stg50,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg50.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg50,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg50 \
     -device scsi-hd,scsi-id=127,lun=16382,id=stg50,bus=virtio_scsi_pci11.0,drive=drive_stg50,write-cache=on \
     -blockdev node-name=file_stg51,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg51.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg51,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg51 \
     -device scsi-hd,scsi-id=254,lun=0,id=stg51,bus=virtio_scsi_pci11.0,drive=drive_stg51,write-cache=on \
     -blockdev node-name=file_stg52,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg52.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg52,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg52 \
     -device scsi-hd,scsi-id=254,lun=8191,id=stg52,bus=virtio_scsi_pci11.0,drive=drive_stg52,write-cache=on \
     -blockdev node-name=file_stg53,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg53.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg53,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg53 \
     -device scsi-hd,scsi-id=254,lun=16382,id=stg53,bus=virtio_scsi_pci11.0,drive=drive_stg53,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-14,port=0xe,addr=0x3.0x6,bus=pcie.0,chassis=15 \
     -device virtio-scsi-pci,id=virtio_scsi_pci12,bus=pcie-root-port-14,addr=0x0,iothread=iothread1 \
     -device pcie-root-port,id=pcie-root-port-15,port=0xf,addr=0x3.0x7,bus=pcie.0,chassis=16 \
     -device virtio-scsi-pci,id=virtio_scsi_pci13,bus=pcie-root-port-15,addr=0x0,iothread=iothread1 \
     -blockdev node-name=file_stg54,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg54.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg54,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg54 \
     -device scsi-hd,scsi-id=0,lun=0,id=stg54,bus=virtio_scsi_pci13.0,drive=drive_stg54,write-cache=on \
     -blockdev node-name=file_stg55,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg55.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg55,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg55 \
     -device scsi-hd,scsi-id=0,lun=8191,id=stg55,bus=virtio_scsi_pci13.0,drive=drive_stg55,write-cache=on \
     -blockdev node-name=file_stg56,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg56.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg56,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg56 \
     -device scsi-hd,scsi-id=0,lun=16382,id=stg56,bus=virtio_scsi_pci13.0,drive=drive_stg56,write-cache=on \
     -blockdev node-name=file_stg57,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg57.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg57,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg57 \
     -device scsi-hd,scsi-id=127,lun=0,id=stg57,bus=virtio_scsi_pci13.0,drive=drive_stg57,write-cache=on \
     -blockdev node-name=file_stg58,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg58.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg58,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg58 \
     -device scsi-hd,scsi-id=127,lun=8191,id=stg58,bus=virtio_scsi_pci13.0,drive=drive_stg58,write-cache=on \
     -blockdev node-name=file_stg59,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg59.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg59,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg59 \
     -device scsi-hd,scsi-id=127,lun=16382,id=stg59,bus=virtio_scsi_pci13.0,drive=drive_stg59,write-cache=on \
     -blockdev node-name=file_stg60,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg60.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg60,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg60 \
     -device scsi-hd,scsi-id=254,lun=0,id=stg60,bus=virtio_scsi_pci13.0,drive=drive_stg60,write-cache=on \
     -blockdev node-name=file_stg61,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg61.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg61,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg61 \
     -device scsi-hd,scsi-id=254,lun=8191,id=stg61,bus=virtio_scsi_pci13.0,drive=drive_stg61,write-cache=on \
     -blockdev node-name=file_stg62,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg62.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg62,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg62 \
     -device scsi-hd,scsi-id=254,lun=16382,id=stg62,bus=virtio_scsi_pci13.0,drive=drive_stg62,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-16,port=0x10,multifunction=on,bus=pcie.0,addr=0x4,chassis=17 \
     -device virtio-net-pci,mac=9a:aa:4a:12:ba:a3,id=idTg2rCG,netdev=id9bdtlh,bus=pcie-root-port-16,addr=0x0  \
     -netdev tap,id=id9bdtlh,vhost=on  \
     -vnc :5  \
    -monitor stdio \
    -qmp tcp:0:5955,server=on,wait=off \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x5,chassis=18





scenario 2 : ovmf+iothread+aio_native +rhel9 , hit issue


file=/home/kvm_autotest_root/images/2090998_OVMF_VARS.fd
[ -f $file ] || {  cp -f /usr/share/OVMF/OVMF_VARS.fd $file; }
/usr/libexec/qemu-kvm \
     -name 'avocado-vt-ovmf'  \
     -sandbox on  \
     -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
     -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
     -blockdev node-name=file_ovmf_vars,driver=file,filename=$file,auto-read-only=on,discard=unmap \
     -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
     -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
     -nodefaults \
     -device VGA,bus=pcie.0,addr=0x2 \
     -m 12G \
     -object memory-backend-ram,size=12G,id=mem-machine_mem  \
     -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
     -cpu 'EPYC-Rome',+kvm_pv_unhalt \
     \
     -chardev socket,id=chardev_serial0,path=/var/tmp/serial-serial0-20220614-012101-usFKQQd2,wait=off,server=on \
     -device isa-serial,id=serial0,chardev=chardev_serial0  \
     \
     -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
     -object iothread,id=iothread0 \
     -object iothread,id=iothread1 \
     -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
     -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
     -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi-ovmf.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
     -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
     -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-3,addr=0x0,iothread=iothread1 \
     -blockdev node-name=file_stg0,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg0.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg0,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg0 \
     -device scsi-hd,scsi-id=0,lun=0,id=stg0,bus=virtio_scsi_pci1.0,drive=drive_stg0,write-cache=on \
     -blockdev node-name=file_stg1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg1.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg1 \
     -device scsi-hd,scsi-id=0,lun=8191,id=stg1,bus=virtio_scsi_pci1.0,drive=drive_stg1,write-cache=on \
 ....
     -blockdev node-name=file_stg62,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg62.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_stg62,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg62 \
     -device scsi-hd,scsi-id=254,lun=16382,id=stg62,bus=virtio_scsi_pci13.0,drive=drive_stg62,write-cache=on \
     -device pcie-root-port,id=pcie-root-port-16,port=0x10,multifunction=on,bus=pcie.0,addr=0x4,chassis=17 \
     -device virtio-net-pci,mac=9a:aa:4a:12:ba:a3,id=idTg2rCG,netdev=id9bdtlh,bus=pcie-root-port-16,addr=0x0  \
     -netdev tap,id=id9bdtlh,vhost=on  \
     -vnc :5  \
    -monitor stdio \
    -qmp tcp:0:5955,server=on,wait=off \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x5,chassis=18


scenario 3 : ovmf+iothread+aio_threads +rhel9 , no issue
scenario 4 : ovmf+iothread+aio_native + rhel8.7 , no issue
scenario 5 : ovmf+aio_native +rhel9 , no issue
Comment 9 Dr. David Alan Gilbert 2022-06-14 08:21:46 UTC 
scenario 1 : seabios +iothread +aio_native+rhel9, no issue
scenario 2 : ovmf+iothread+aio_native +rhel9 , hit issue
scenario 3 : ovmf+iothread+aio_threads +rhel9 , no issue

so either switching to seabios, OR switching to aio_thread fixes it
OR a different host.
Hmm.
Maybe it's just a race condition here and you just have to be lucky to hit it; but given the aio stuff,
maybe it's worth asking stefanha.
Comment 10 Stefan Hajnoczi 2022-07-05 13:38:47 UTC 
This might be a duplicate of bz2092788, which was fixed in qemu-kvm-7.0.0-7.el9. Please retry with qemu-kvm-7.0.0-7.el9 or later. Thanks!
Comment 11 qing.wang 2022-07-08 08:29:39 UTC 
This issue still can be reproduecd on my Host.
Red Hat Enterprise Linux release 9.1 Beta (Plow)
5.14.0-124.el9.x86_64
qemu-kvm-7.0.0-8.el9.x86_64 --> qemu-kvm-7.0.0-7.el9.x86_64  same result
seabios-bin-1.16.0-3.el9.noarch
edk2-ovmf-20220526git16779ede2d36-2.el9.noarch
Comment 12 Stefan Hajnoczi 2022-07-13 14:09:41 UTC 
(In reply to qing.wang from comment #11)
> This issue still can be reproduecd on my Host.
> Red Hat Enterprise Linux release 9.1 Beta (Plow)
> 5.14.0-124.el9.x86_64
> qemu-kvm-7.0.0-8.el9.x86_64 --> qemu-kvm-7.0.0-7.el9.x86_64  same result
> seabios-bin-1.16.0-3.el9.noarch
> edk2-ovmf-20220526git16779ede2d36-2.el9.noarch

Thanks for checking this. In that case I suggest monitoring the virtio-scsi activity to understand what is going on. The guest kernel should be sending SCSI INQUIRY commands to discover the LUNs. For some reason this becomes stuck. Things to look at:
- Guest thread backtraces (kernel magic SysRq-t)
- QEMU thread backtraces
- What SCSI requests are in flight in QEMU's SCSI emulation?
- Are there pending qcow2 and image file I/O requests in QEMU?

I will be on PTO from July 18-29 so I can't answer these questions soon. Does someone else want to give it a shot? Maybe Paolo Bonzini, Kevin Wolf, Hanna Reitz, or Emanuele Esposito?
Comment 23 Stefan Hajnoczi 2022-12-15 15:59:33 UTC 
Is it possible to provide access to a running VM that has hit the problem? I sshed into the two hosts mentioned in the comments, but didn't see any VMs.

Thanks!
Comment 25 Vivek Goyal 2023-03-08 22:15:02 UTC 
Hi Stefan,

Is there any update on this bug.
Comment 26 Vivek Goyal 2023-03-08 22:16:24 UTC 
This bug currently is listed as "high" priority and severity. Is that still the case or should we lower the priority to medium instead.
Comment 27 Stefan Hajnoczi 2023-03-15 14:27:11 UTC 
Reducing priority to medium since this issue involves a specific configuration (many disks) and has not been reported by users.
Comment 29 Stefan Hajnoczi 2023-03-15 20:47:15 UTC 
Does this bug reproduce when there is only 1 virtio-scsi-pci device? The guest kernel's SCSI scanning code has global state that is shared by all SCSI adapters and I wonder if there is a race condition that led to do_scan_async() waiting for a long time.
Comment 32 qing.wang 2023-03-18 04:07:08 UTC 
Hit same issue

Red Hat Enterprise Linux release 9.2 Beta (Plow)
5.14.0-284.2.1.el9_2.x86_64
qemu-kvm-7.2.0-12.el9_2.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20221207gitfff6d81270b5-8.el9_2.noarch
libvirt-9.0.0-7.el9.x86_64
virtio-win-prewhql-0.1-234.iso

automation log

http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section2/images_backup/qbugs/2090998/2023-03-18/
Comment 38 Stefan Hajnoczi 2023-03-22 14:05:12 UTC 
Created attachment 1952802 [details]
GDB Python script to check virtio-scsi-pci devices
Comment 39 Stefan Hajnoczi 2023-03-22 14:06:17 UTC 
Created attachment 1952803 [details]
GDB Python script to print AioHandler in an IOThread
Comment 41 Stefan Hajnoczi 2023-03-22 14:58:22 UTC 
Patch posted upstream:
https://patchew.org/QEMU/20230322145521.1294954-1-stefanha@redhat.com/
Comment 46 Yanan Fu 2023-04-24 10:33:52 UTC 
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.
Comment 49 qing.wang 2023-04-27 05:25:51 UTC 
Passed test on
Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-303.el9.x86_64
qemu-kvm-8.0.0-1.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230301gitf80f052277c8-2.el9.noarch
libvirt-9.0.0-10.el9_2.x86_64
virtio-win-prewhql-0.1-235.iso


python ConfigTest.py --testcase=multi_disk.virtio_scsi_variants.multi_bus_scsiid_lun.q35 --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=RHEL.9.2.0 --driveformat=virtio_scsi --nicmodel=virtio_net --imageformat=qcow2 --machines=q35 --customsparams="vm_mem_limit = 12G\nimage_aio=native" --clone=no --firmware=ovmf --nrepeat=30 --netdst=virbr0
Comment 56 errata-xmlrpc 2023-11-07 08:26:38 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368
Note You need to log in before you can comment on or make changes to this bug.