Bug 2090998
| Summary: | [qemu-kvm] rhel guest failed boot with multi disks on error Failed to start udev Wait for Complete Device Initialization | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | qing.wang <qinwang> | ||||||
| Component: | qemu-kvm | Assignee: | Stefan Hajnoczi <stefanha> | ||||||
| qemu-kvm sub component: | Storage | QA Contact: | qing.wang <qinwang> | ||||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||||
| Severity: | high | ||||||||
| Priority: | medium | CC: | coli, jferlan, jinzhao, juzhang, kwolf, lijin, mdeng, meili, mrezanin, pbonzini, qzhang, stefanha, vgoyal, virt-maint, xuwei, ymankad | ||||||
| Version: | 9.1 | Keywords: | CustomerScenariosInitiative, Triaged, ZStream | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | All | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | qemu-kvm-8.0.0-1.el9 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 2211923 (view as bug list) | Environment: | |||||||
| Last Closed: | 2023-11-07 08:26:38 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 2211923 | ||||||||
| Attachments: |
|
||||||||
Passed on other AMD host, Red Hat Enterprise Linux release 8.6 (Ootpa) 4.18.0-372.10.1.el8_6.x86_64 qemu-kvm-6.2.0-11.module+el8.6.0+14712+f96656d3.x86_64 seabios-bin-1.15.0-2.module+el8.6.0+14757+c25ee005.noarch edk2-ovmf-20220126gitbb1bba3d77-2.el8.noarch python3-libvirt-8.0.0-1.module+el8.6.0+13896+a8fa8f67.x86_64 virtio-win-prewhql-0.1-219.iso Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 NUMA node(s): 1 Vendor ID: AuthenticAMD BIOS Vendor ID: AMD CPU family: 23 Model: 49 Model name: AMD EPYC 7302P 16-Core Processor BIOS Model name: AMD EPYC 7302P 16-Core Processor Stepping: 0 CPU MHz: 2994.247 BogoMIPS: 5988.49 Virtualization: AMD-V L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 16384K NUMA node0 CPU(s): 0-31 It looks like regression issue ? (In reply to qing.wang from comment #3) > Passed on other AMD host, > > Red Hat Enterprise Linux release 8.6 (Ootpa) > 4.18.0-372.10.1.el8_6.x86_64 > qemu-kvm-6.2.0-11.module+el8.6.0+14712+f96656d3.x86_64 > seabios-bin-1.15.0-2.module+el8.6.0+14757+c25ee005.noarch > edk2-ovmf-20220126gitbb1bba3d77-2.el8.noarch > python3-libvirt-8.0.0-1.module+el8.6.0+13896+a8fa8f67.x86_64 > virtio-win-prewhql-0.1-219.iso > > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 32 > On-line CPU(s) list: 0-31 > Thread(s) per core: 2 > Core(s) per socket: 16 > Socket(s): 1 > NUMA node(s): 1 > Vendor ID: AuthenticAMD > BIOS Vendor ID: AMD > CPU family: 23 > Model: 49 > Model name: AMD EPYC 7302P 16-Core Processor > BIOS Model name: AMD EPYC 7302P 16-Core Processor > Stepping: 0 > CPU MHz: 2994.247 > BogoMIPS: 5988.49 > Virtualization: AMD-V > L1d cache: 32K > L1i cache: 32K > L2 cache: 512K > L3 cache: 16384K > NUMA node0 CPU(s): 0-31 > > > It looks like regression issue ? Can you qualify that further? Because it looks like the HOST cpu is relevant to triggering the issue, but I'm not sure we are seeing evidence of regression in the software (i.e., are you able to pass the test on a hardware that demonstrably fails, but using an older version for OVMF?) Gerd, this is an oddball, can you take it? -Klaus > > It looks like regression issue ?
>
> Can you qualify that further? Because it looks like the HOST cpu is relevant
> to triggering the issue, but I'm not sure we are seeing evidence of
> regression in the software (i.e., are you able to pass the test on a
> hardware that demonstrably fails, but using an older version for OVMF?)
Guest kernel apparently waiting for disk i/o. Not obvious what the root cause is.
Given the same test works fine on another amd box it could be bad hardware.
So, yes, we need more info to qualify whenever this is a regression or not.
Does the test work (on the same hardware) reliable with older versions?
Can you try run 'memtest' on the machine where the test fails?
I run same testing on host as #0 with different qemu.(ovmf using same) Not issue: Red Hat Enterprise Linux release 9.0 (Plow) 5.14.0-70.15.1.el9_0.x86_64 qemu-kvm-6.2.0-11.el9_0.3.x86_64 seabios-bin-1.15.0-1.el9.noarch edk2-ovmf-20220126gitbb1bba3d77-3.el9.noarch virtio-win-prewhql-0.1-221.iso Hit issue: Red Hat Enterprise Linux release 9.0 (Plow) 5.14.0-70.15.1.el9_0.x86_64 qemu-kvm-7.0.0-5.el9.x86_64 seabios-bin-1.15.0-1.el9.noarch edk2-ovmf-20220126gitbb1bba3d77-3.el9.noarch virtio-win-prewhql-0.1-221.iso > I run same testing on host as #0 with different qemu.(ovmf using same) > Not issue: > qemu-kvm-6.2.0-11.el9_0.3.x86_64 > Hit issue: > qemu-kvm-7.0.0-5.el9.x86_64 So, the qemu rebase apparently broke this. Why is this tagged as ovmf issue? Does the same configuration with seabios (and qemu 7.0) work fine? (In reply to Gerd Hoffmann from comment #7) > > I run same testing on host as #0 with different qemu.(ovmf using same) > > > Not issue: > > qemu-kvm-6.2.0-11.el9_0.3.x86_64 > > > Hit issue: > > qemu-kvm-7.0.0-5.el9.x86_64 > > So, the qemu rebase apparently broke this. > Why is this tagged as ovmf issue? > Does the same configuration with seabios (and qemu 7.0) work fine? This issue related to multi factors:iothread , aio ,guest,ovmf scenario 1 : seabios +iothread +aio_native+rhel9, no issue /usr/libexec/qemu-kvm \ -name 'avocado-vt-bios' \ -sandbox on \ -machine q35,memory-backend=mem-machine_mem \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 12G \ -object memory-backend-ram,size=12G,id=mem-machine_mem \ -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \ -cpu 'EPYC-Rome',+kvm_pv_unhalt \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -object iothread,id=iothread0 \ -object iothread,id=iothread1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi-bios.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-3,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg0,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg0.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg0,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg0 \ -device scsi-hd,scsi-id=0,lun=0,id=stg0,bus=virtio_scsi_pci1.0,drive=drive_stg0,write-cache=on \ -blockdev node-name=file_stg1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg1.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg1 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg1,bus=virtio_scsi_pci1.0,drive=drive_stg1,write-cache=on \ -blockdev node-name=file_stg2,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg2.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg2,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg2 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg2,bus=virtio_scsi_pci1.0,drive=drive_stg2,write-cache=on \ -blockdev node-name=file_stg3,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg3.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg3,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg3 \ -device scsi-hd,scsi-id=127,lun=0,id=stg3,bus=virtio_scsi_pci1.0,drive=drive_stg3,write-cache=on \ -blockdev node-name=file_stg4,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg4.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg4,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg4 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg4,bus=virtio_scsi_pci1.0,drive=drive_stg4,write-cache=on \ -blockdev node-name=file_stg5,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg5.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg5,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg5 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg5,bus=virtio_scsi_pci1.0,drive=drive_stg5,write-cache=on \ -blockdev node-name=file_stg6,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg6.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg6,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg6 \ -device scsi-hd,scsi-id=254,lun=0,id=stg6,bus=virtio_scsi_pci1.0,drive=drive_stg6,write-cache=on \ -blockdev node-name=file_stg7,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg7.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg7,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg7 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg7,bus=virtio_scsi_pci1.0,drive=drive_stg7,write-cache=on \ -blockdev node-name=file_stg8,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg8.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg8,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg8 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg8,bus=virtio_scsi_pci1.0,drive=drive_stg8,write-cache=on \ -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \ -device virtio-scsi-pci,id=virtio_scsi_pci2,bus=pcie-root-port-4,addr=0x0,iothread=iothread0 \ -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \ -device virtio-scsi-pci,id=virtio_scsi_pci3,bus=pcie-root-port-5,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_stg9,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg9.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg9,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg9 \ -device scsi-hd,scsi-id=0,lun=0,id=stg9,bus=virtio_scsi_pci3.0,drive=drive_stg9,write-cache=on \ -blockdev node-name=file_stg10,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg10.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg10,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg10 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg10,bus=virtio_scsi_pci3.0,drive=drive_stg10,write-cache=on \ -blockdev node-name=file_stg11,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg11.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg11,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg11 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg11,bus=virtio_scsi_pci3.0,drive=drive_stg11,write-cache=on \ -blockdev node-name=file_stg12,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg12.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg12,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg12 \ -device scsi-hd,scsi-id=127,lun=0,id=stg12,bus=virtio_scsi_pci3.0,drive=drive_stg12,write-cache=on \ -blockdev node-name=file_stg13,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg13.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg13,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg13 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg13,bus=virtio_scsi_pci3.0,drive=drive_stg13,write-cache=on \ -blockdev node-name=file_stg14,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg14.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg14,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg14 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg14,bus=virtio_scsi_pci3.0,drive=drive_stg14,write-cache=on \ -blockdev node-name=file_stg15,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg15.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg15,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg15 \ -device scsi-hd,scsi-id=254,lun=0,id=stg15,bus=virtio_scsi_pci3.0,drive=drive_stg15,write-cache=on \ -blockdev node-name=file_stg16,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg16.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg16,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg16 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg16,bus=virtio_scsi_pci3.0,drive=drive_stg16,write-cache=on \ -blockdev node-name=file_stg17,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg17.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg17,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg17 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg17,bus=virtio_scsi_pci3.0,drive=drive_stg17,write-cache=on \ -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x1.0x6,bus=pcie.0,chassis=7 \ -device virtio-scsi-pci,id=virtio_scsi_pci4,bus=pcie-root-port-6,addr=0x0,iothread=iothread1 \ -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x1.0x7,bus=pcie.0,chassis=8 \ -device virtio-scsi-pci,id=virtio_scsi_pci5,bus=pcie-root-port-7,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg18,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg18.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg18,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg18 \ -device scsi-hd,scsi-id=0,lun=0,id=stg18,bus=virtio_scsi_pci5.0,drive=drive_stg18,write-cache=on \ -blockdev node-name=file_stg19,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg19.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg19,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg19 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg19,bus=virtio_scsi_pci5.0,drive=drive_stg19,write-cache=on \ -blockdev node-name=file_stg20,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg20.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg20,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg20 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg20,bus=virtio_scsi_pci5.0,drive=drive_stg20,write-cache=on \ -blockdev node-name=file_stg21,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg21.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg21,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg21 \ -device scsi-hd,scsi-id=127,lun=0,id=stg21,bus=virtio_scsi_pci5.0,drive=drive_stg21,write-cache=on \ -blockdev node-name=file_stg22,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg22.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg22,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg22 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg22,bus=virtio_scsi_pci5.0,drive=drive_stg22,write-cache=on \ -blockdev node-name=file_stg23,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg23.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg23,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg23 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg23,bus=virtio_scsi_pci5.0,drive=drive_stg23,write-cache=on \ -blockdev node-name=file_stg24,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg24.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg24,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg24 \ -device scsi-hd,scsi-id=254,lun=0,id=stg24,bus=virtio_scsi_pci5.0,drive=drive_stg24,write-cache=on \ -blockdev node-name=file_stg25,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg25.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg25,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg25 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg25,bus=virtio_scsi_pci5.0,drive=drive_stg25,write-cache=on \ -blockdev node-name=file_stg26,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg26.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg26,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg26 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg26,bus=virtio_scsi_pci5.0,drive=drive_stg26,write-cache=on \ -device pcie-root-port,id=pcie-root-port-8,port=0x8,multifunction=on,bus=pcie.0,addr=0x3,chassis=9 \ -device virtio-scsi-pci,id=virtio_scsi_pci6,bus=pcie-root-port-8,addr=0x0,iothread=iothread0 \ -device pcie-root-port,id=pcie-root-port-9,port=0x9,addr=0x3.0x1,bus=pcie.0,chassis=10 \ -device virtio-scsi-pci,id=virtio_scsi_pci7,bus=pcie-root-port-9,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_stg27,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg27.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg27,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg27 \ -device scsi-hd,scsi-id=0,lun=0,id=stg27,bus=virtio_scsi_pci7.0,drive=drive_stg27,write-cache=on \ -blockdev node-name=file_stg28,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg28.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg28,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg28 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg28,bus=virtio_scsi_pci7.0,drive=drive_stg28,write-cache=on \ -blockdev node-name=file_stg29,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg29.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg29,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg29 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg29,bus=virtio_scsi_pci7.0,drive=drive_stg29,write-cache=on \ -blockdev node-name=file_stg30,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg30.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg30,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg30 \ -device scsi-hd,scsi-id=127,lun=0,id=stg30,bus=virtio_scsi_pci7.0,drive=drive_stg30,write-cache=on \ -blockdev node-name=file_stg31,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg31.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg31,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg31 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg31,bus=virtio_scsi_pci7.0,drive=drive_stg31,write-cache=on \ -blockdev node-name=file_stg32,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg32.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg32,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg32 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg32,bus=virtio_scsi_pci7.0,drive=drive_stg32,write-cache=on \ -blockdev node-name=file_stg33,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg33.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg33,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg33 \ -device scsi-hd,scsi-id=254,lun=0,id=stg33,bus=virtio_scsi_pci7.0,drive=drive_stg33,write-cache=on \ -blockdev node-name=file_stg34,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg34.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg34,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg34 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg34,bus=virtio_scsi_pci7.0,drive=drive_stg34,write-cache=on \ -blockdev node-name=file_stg35,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg35.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg35,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg35 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg35,bus=virtio_scsi_pci7.0,drive=drive_stg35,write-cache=on \ -device pcie-root-port,id=pcie-root-port-10,port=0xa,addr=0x3.0x2,bus=pcie.0,chassis=11 \ -device virtio-scsi-pci,id=virtio_scsi_pci8,bus=pcie-root-port-10,addr=0x0,iothread=iothread1 \ -device pcie-root-port,id=pcie-root-port-11,port=0xb,addr=0x3.0x3,bus=pcie.0,chassis=12 \ -device virtio-scsi-pci,id=virtio_scsi_pci9,bus=pcie-root-port-11,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg36,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg36.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg36,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg36 \ -device scsi-hd,scsi-id=0,lun=0,id=stg36,bus=virtio_scsi_pci9.0,drive=drive_stg36,write-cache=on \ -blockdev node-name=file_stg37,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg37.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg37,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg37 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg37,bus=virtio_scsi_pci9.0,drive=drive_stg37,write-cache=on \ -blockdev node-name=file_stg38,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg38.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg38,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg38 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg38,bus=virtio_scsi_pci9.0,drive=drive_stg38,write-cache=on \ -blockdev node-name=file_stg39,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg39.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg39,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg39 \ -device scsi-hd,scsi-id=127,lun=0,id=stg39,bus=virtio_scsi_pci9.0,drive=drive_stg39,write-cache=on \ -blockdev node-name=file_stg40,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg40.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg40,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg40 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg40,bus=virtio_scsi_pci9.0,drive=drive_stg40,write-cache=on \ -blockdev node-name=file_stg41,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg41.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg41,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg41 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg41,bus=virtio_scsi_pci9.0,drive=drive_stg41,write-cache=on \ -blockdev node-name=file_stg42,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg42.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg42,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg42 \ -device scsi-hd,scsi-id=254,lun=0,id=stg42,bus=virtio_scsi_pci9.0,drive=drive_stg42,write-cache=on \ -blockdev node-name=file_stg43,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg43.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg43,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg43 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg43,bus=virtio_scsi_pci9.0,drive=drive_stg43,write-cache=on \ -blockdev node-name=file_stg44,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg44.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg44,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg44 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg44,bus=virtio_scsi_pci9.0,drive=drive_stg44,write-cache=on \ -device pcie-root-port,id=pcie-root-port-12,port=0xc,addr=0x3.0x4,bus=pcie.0,chassis=13 \ -device virtio-scsi-pci,id=virtio_scsi_pci10,bus=pcie-root-port-12,addr=0x0,iothread=iothread0 \ -device pcie-root-port,id=pcie-root-port-13,port=0xd,addr=0x3.0x5,bus=pcie.0,chassis=14 \ -device virtio-scsi-pci,id=virtio_scsi_pci11,bus=pcie-root-port-13,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_stg45,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg45.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg45,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg45 \ -device scsi-hd,scsi-id=0,lun=0,id=stg45,bus=virtio_scsi_pci11.0,drive=drive_stg45,write-cache=on \ -blockdev node-name=file_stg46,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg46.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg46,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg46 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg46,bus=virtio_scsi_pci11.0,drive=drive_stg46,write-cache=on \ -blockdev node-name=file_stg47,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg47.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg47,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg47 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg47,bus=virtio_scsi_pci11.0,drive=drive_stg47,write-cache=on \ -blockdev node-name=file_stg48,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg48.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg48,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg48 \ -device scsi-hd,scsi-id=127,lun=0,id=stg48,bus=virtio_scsi_pci11.0,drive=drive_stg48,write-cache=on \ -blockdev node-name=file_stg49,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg49.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg49,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg49 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg49,bus=virtio_scsi_pci11.0,drive=drive_stg49,write-cache=on \ -blockdev node-name=file_stg50,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg50.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg50,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg50 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg50,bus=virtio_scsi_pci11.0,drive=drive_stg50,write-cache=on \ -blockdev node-name=file_stg51,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg51.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg51,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg51 \ -device scsi-hd,scsi-id=254,lun=0,id=stg51,bus=virtio_scsi_pci11.0,drive=drive_stg51,write-cache=on \ -blockdev node-name=file_stg52,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg52.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg52,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg52 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg52,bus=virtio_scsi_pci11.0,drive=drive_stg52,write-cache=on \ -blockdev node-name=file_stg53,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg53.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg53,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg53 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg53,bus=virtio_scsi_pci11.0,drive=drive_stg53,write-cache=on \ -device pcie-root-port,id=pcie-root-port-14,port=0xe,addr=0x3.0x6,bus=pcie.0,chassis=15 \ -device virtio-scsi-pci,id=virtio_scsi_pci12,bus=pcie-root-port-14,addr=0x0,iothread=iothread1 \ -device pcie-root-port,id=pcie-root-port-15,port=0xf,addr=0x3.0x7,bus=pcie.0,chassis=16 \ -device virtio-scsi-pci,id=virtio_scsi_pci13,bus=pcie-root-port-15,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg54,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg54.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg54,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg54 \ -device scsi-hd,scsi-id=0,lun=0,id=stg54,bus=virtio_scsi_pci13.0,drive=drive_stg54,write-cache=on \ -blockdev node-name=file_stg55,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg55.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg55,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg55 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg55,bus=virtio_scsi_pci13.0,drive=drive_stg55,write-cache=on \ -blockdev node-name=file_stg56,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg56.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg56,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg56 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg56,bus=virtio_scsi_pci13.0,drive=drive_stg56,write-cache=on \ -blockdev node-name=file_stg57,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg57.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg57,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg57 \ -device scsi-hd,scsi-id=127,lun=0,id=stg57,bus=virtio_scsi_pci13.0,drive=drive_stg57,write-cache=on \ -blockdev node-name=file_stg58,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg58.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg58,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg58 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg58,bus=virtio_scsi_pci13.0,drive=drive_stg58,write-cache=on \ -blockdev node-name=file_stg59,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg59.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg59,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg59 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg59,bus=virtio_scsi_pci13.0,drive=drive_stg59,write-cache=on \ -blockdev node-name=file_stg60,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg60.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg60,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg60 \ -device scsi-hd,scsi-id=254,lun=0,id=stg60,bus=virtio_scsi_pci13.0,drive=drive_stg60,write-cache=on \ -blockdev node-name=file_stg61,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg61.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg61,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg61 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg61,bus=virtio_scsi_pci13.0,drive=drive_stg61,write-cache=on \ -blockdev node-name=file_stg62,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg62.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg62,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg62 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg62,bus=virtio_scsi_pci13.0,drive=drive_stg62,write-cache=on \ -device pcie-root-port,id=pcie-root-port-16,port=0x10,multifunction=on,bus=pcie.0,addr=0x4,chassis=17 \ -device virtio-net-pci,mac=9a:aa:4a:12:ba:a3,id=idTg2rCG,netdev=id9bdtlh,bus=pcie-root-port-16,addr=0x0 \ -netdev tap,id=id9bdtlh,vhost=on \ -vnc :5 \ -monitor stdio \ -qmp tcp:0:5955,server=on,wait=off \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x5,chassis=18 scenario 2 : ovmf+iothread+aio_native +rhel9 , hit issue file=/home/kvm_autotest_root/images/2090998_OVMF_VARS.fd [ -f $file ] || { cp -f /usr/share/OVMF/OVMF_VARS.fd $file; } /usr/libexec/qemu-kvm \ -name 'avocado-vt-ovmf' \ -sandbox on \ -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \ -blockdev node-name=file_ovmf_vars,driver=file,filename=$file,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \ -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 12G \ -object memory-backend-ram,size=12G,id=mem-machine_mem \ -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \ -cpu 'EPYC-Rome',+kvm_pv_unhalt \ \ -chardev socket,id=chardev_serial0,path=/var/tmp/serial-serial0-20220614-012101-usFKQQd2,wait=off,server=on \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -object iothread,id=iothread0 \ -object iothread,id=iothread1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi-ovmf.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-3,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg0,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg0.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg0,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg0 \ -device scsi-hd,scsi-id=0,lun=0,id=stg0,bus=virtio_scsi_pci1.0,drive=drive_stg0,write-cache=on \ -blockdev node-name=file_stg1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg1.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg1 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg1,bus=virtio_scsi_pci1.0,drive=drive_stg1,write-cache=on \ .... -blockdev node-name=file_stg62,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg62.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg62,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg62 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg62,bus=virtio_scsi_pci13.0,drive=drive_stg62,write-cache=on \ -device pcie-root-port,id=pcie-root-port-16,port=0x10,multifunction=on,bus=pcie.0,addr=0x4,chassis=17 \ -device virtio-net-pci,mac=9a:aa:4a:12:ba:a3,id=idTg2rCG,netdev=id9bdtlh,bus=pcie-root-port-16,addr=0x0 \ -netdev tap,id=id9bdtlh,vhost=on \ -vnc :5 \ -monitor stdio \ -qmp tcp:0:5955,server=on,wait=off \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x5,chassis=18 scenario 3 : ovmf+iothread+aio_threads +rhel9 , no issue scenario 4 : ovmf+iothread+aio_native + rhel8.7 , no issue scenario 5 : ovmf+aio_native +rhel9 , no issue scenario 1 : seabios +iothread +aio_native+rhel9, no issue scenario 2 : ovmf+iothread+aio_native +rhel9 , hit issue scenario 3 : ovmf+iothread+aio_threads +rhel9 , no issue so either switching to seabios, OR switching to aio_thread fixes it OR a different host. Hmm. Maybe it's just a race condition here and you just have to be lucky to hit it; but given the aio stuff, maybe it's worth asking stefanha. This might be a duplicate of bz2092788, which was fixed in qemu-kvm-7.0.0-7.el9. Please retry with qemu-kvm-7.0.0-7.el9 or later. Thanks! This issue still can be reproduecd on my Host. Red Hat Enterprise Linux release 9.1 Beta (Plow) 5.14.0-124.el9.x86_64 qemu-kvm-7.0.0-8.el9.x86_64 --> qemu-kvm-7.0.0-7.el9.x86_64 same result seabios-bin-1.16.0-3.el9.noarch edk2-ovmf-20220526git16779ede2d36-2.el9.noarch (In reply to qing.wang from comment #11) > This issue still can be reproduecd on my Host. > Red Hat Enterprise Linux release 9.1 Beta (Plow) > 5.14.0-124.el9.x86_64 > qemu-kvm-7.0.0-8.el9.x86_64 --> qemu-kvm-7.0.0-7.el9.x86_64 same result > seabios-bin-1.16.0-3.el9.noarch > edk2-ovmf-20220526git16779ede2d36-2.el9.noarch Thanks for checking this. In that case I suggest monitoring the virtio-scsi activity to understand what is going on. The guest kernel should be sending SCSI INQUIRY commands to discover the LUNs. For some reason this becomes stuck. Things to look at: - Guest thread backtraces (kernel magic SysRq-t) - QEMU thread backtraces - What SCSI requests are in flight in QEMU's SCSI emulation? - Are there pending qcow2 and image file I/O requests in QEMU? I will be on PTO from July 18-29 so I can't answer these questions soon. Does someone else want to give it a shot? Maybe Paolo Bonzini, Kevin Wolf, Hanna Reitz, or Emanuele Esposito? Is it possible to provide access to a running VM that has hit the problem? I sshed into the two hosts mentioned in the comments, but didn't see any VMs. Thanks! Hi Stefan, Is there any update on this bug. This bug currently is listed as "high" priority and severity. Is that still the case or should we lower the priority to medium instead. Reducing priority to medium since this issue involves a specific configuration (many disks) and has not been reported by users. Does this bug reproduce when there is only 1 virtio-scsi-pci device? The guest kernel's SCSI scanning code has global state that is shared by all SCSI adapters and I wonder if there is a race condition that led to do_scan_async() waiting for a long time. Hit same issue Red Hat Enterprise Linux release 9.2 Beta (Plow) 5.14.0-284.2.1.el9_2.x86_64 qemu-kvm-7.2.0-12.el9_2.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20221207gitfff6d81270b5-8.el9_2.noarch libvirt-9.0.0-7.el9.x86_64 virtio-win-prewhql-0.1-234.iso automation log http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section2/images_backup/qbugs/2090998/2023-03-18/ Created attachment 1952802 [details]
GDB Python script to check virtio-scsi-pci devices
Created attachment 1952803 [details]
GDB Python script to print AioHandler in an IOThread
Patch posted upstream: https://patchew.org/QEMU/20230322145521.1294954-1-stefanha@redhat.com/ QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass. Passed test on Red Hat Enterprise Linux release 9.3 Beta (Plow) 5.14.0-303.el9.x86_64 qemu-kvm-8.0.0-1.el9.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20230301gitf80f052277c8-2.el9.noarch libvirt-9.0.0-10.el9_2.x86_64 virtio-win-prewhql-0.1-235.iso python ConfigTest.py --testcase=multi_disk.virtio_scsi_variants.multi_bus_scsiid_lun.q35 --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=RHEL.9.2.0 --driveformat=virtio_scsi --nicmodel=virtio_net --imageformat=qcow2 --machines=q35 --customsparams="vm_mem_limit = 12G\nimage_aio=native" --clone=no --firmware=ovmf --nrepeat=30 --netdst=virbr0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6368 |
Description of problem: Boot guest with multi disk and bus on AMD cpu host. It failed on booting and kernel crash 2022-05-27 04:17:13: [K[ [0;31m*[0;1;31m*[0m] (2 of 2) A start job is running fordroot.device (1min 56s / no limit) 2022-05-27 04:17:13: [K[ [0;31m*[0m] (2 of 2) A start job is running fordroot.device (1min 57s / no limit) 2022-05-27 04:17:14: [K[ [0;31m*[0;1;31m*[0m] (1 of 2) A start job is running fore Initialization (1min 57s / 3min) 2022-05-27 04:17:14: [K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] (1 of 2) A start job is running fore Initialization (1min 58s / 3min) 2022-05-27 04:17:15: [K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] (1 of 2) A start job is running fore Initialization (1min 58s / 3min) 2022-05-27 04:17:15: [K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] (2 of 2) A start job is running fordroot.device (1min 59s / no limit) 2022-05-27 04:17:16: [K[[0;31m*[0;1;31m*[0m[0;31m* [0m] (2 of 2) A start job is running fordroot.device (1min 59s / no limit) 2022-05-27 04:17:16: [K[[0;1;31m*[0m[0;31m* [0m] (2 of 2) A start job is running for5\x2droot.device (2min / no limit) 2022-05-27 04:17:17: [K[[0m[0;31m* [0m] (1 of 2) A start job is running forevice Initialization (2min / 3min)[K[[0;1;31mFAILED[0m] Failed to start udev Wait for Complete Device Initialization. 2022-05-27 04:17:17: See 'systemctl status systemd-udev-settle.service' for details. 2022-05-27 04:17:17: Starting Device-Mapper Multipath Device Controller... 2022-05-27 04:17:17: [[0;32m OK [0m] Started Device-Mapper Multipath Device Controller. 2022-05-27 04:17:17: [[0;32m OK [0m] Reached target Local File Systems (Pre). 2022-05-27 04:17:17: [[0;32m OK [0m] Reached target Local File Systems. 2022-05-27 04:17:17: Starting Create Volatile Files and Directories... 2022-05-27 04:17:17: Starting Open-iSCSI... 2022-05-27 04:17:17: [[0;32m OK [0m] Started Open-iSCSI. 2022-05-27 04:17:17: [[0;32m OK [0m] Started Create Volatile Files and Directories. 2022-05-27 04:17:17: [[0;32m OK [0m] Reached target System Initialization. 2022-05-27 04:17:17: [[0;32m OK [0m] Reached target Basic System. 2022-05-27 04:17:17: Starting dracut initqueue hook... 2022-05-27 04:17:17: [ 123.304632] TECH PREVIEW: NVMe/TCP may not be fully supported. 2022-05-27 04:17:17: [ 123.304632] Please review provided documentation for limitations. 2022-05-27 04:19:20: [ 246.751028] INFO: task kworker/u32:1:108 blocked for more than 120 seconds. 2022-05-27 04:19:20: [ 246.752331] Tainted: G X --------- - - 4.18.0-372.9.1.el8.x86_64 #1 2022-05-27 04:19:20: [ 246.753538] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2022-05-27 04:19:20: [ 246.754708] task:kworker/u32:1 state:D stack: 0 pid: 108 ppid: 2 flags:0x80004000 2022-05-27 04:19:20: [ 246.755941] Workqueue: events_unbound async_run_entry_fn 2022-05-27 04:19:20: [ 246.756902] Call Trace: 2022-05-27 04:19:20: [ 246.757604] __schedule+0x2d1/0x830 2022-05-27 04:19:20: [ 246.758407] ? kfree+0x1b0/0x1f0 2022-05-27 04:19:20: [ 246.759179] schedule+0x35/0xa0 2022-05-27 04:19:20: [ 246.759927] schedule_timeout+0x274/0x300 2022-05-27 04:19:20: [ 246.760754] wait_for_completion+0x96/0x100 2022-05-27 04:19:20: [ 246.761604] do_scan_async+0x48/0x150 2022-05-27 04:19:20: [ 246.762391] async_run_entry_fn+0x39/0x160 2022-05-27 04:19:20: [ 246.763208] process_one_work+0x1a7/0x360 2022-05-27 04:19:20: [ 246.764017] worker_thread+0x30/0x390 2022-05-27 04:19:20: [ 246.764781] ? create_worker+0x1a0/0x1a0 2022-05-27 04:19:20: [ 246.765575] kthread+0x10a/0x120 2022-05-27 04:19:20: [ 246.766308] ? set_kthread_struct+0x40/0x40 2022-05-27 04:19:20: [ 246.767128] ret_from_fork+0x35/0x40 2022-05-27 04:19:20: [ 246.767885] INFO: task kworker/u32:2:109 blocked for more than 120 seconds. 2022-05-27 04:19:20: [ 246.768950] Tainted: G X --------- - - 4.18.0-372.9.1.el8.x86_64 #1 2022-05-27 04:19:20: [ 246.770120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2022-05-27 04:19:20: [ 246.771273] task:kworker/u32:2 state:D stack: 0 pid: 109 ppid: 2 flags:0x80004000 2022-05-27 04:19:20: [ 246.772487] Workqueue: events_unbound async_run_entry_fn 2022-05-27 04:19:20: [ 246.773433] Call Trace: 2022-05-27 04:19:20: [ 246.774127] __schedule+0x2d1/0x830 2022-05-27 04:19:20: [ 246.774908] ? kfree+0x1b0/0x1f0 2022-05-27 04:19:20: [ 246.775666] schedule+0x35/0xa0 2022-05-27 04:19:20: [ 246.776420] schedule_timeout+0x274/0x300 2022-05-27 04:19:20: [ 246.777259] ? __switch_to_asm+0x41/0x70 2022-05-27 04:19:20: [ 246.778084] ? reschedule_interrupt+0xa/0x20 2022-05-27 04:19:20: [ 246.778926] wait_for_completion+0x96/0x100 2022-05-27 04:19:20: [ 246.779764] do_scan_async+0x48/0x150 2022-05-27 04:19:20: [ 246.780550] async_run_entry_fn+0x39/0x160 2022-05-27 04:19:20: [ 246.781368] process_one_work+0x1a7/0x360 2022-05-27 04:19:20: [ 246.782171] worker_thread+0x30/0x390 2022-05-27 04:19:20: [ 246.782926] ? create_worker+0x1a0/0x1a0 2022-05-27 04:19:20: [ 246.783709] kthread+0x10a/0x120 2022-05-27 04:19:20: [ 246.784428] ? set_kthread_struct+0x40/0x40 2022-05-27 04:19:20: [ 246.785229] ret_from_fork+0x35/0x40 2022-05-27 04:19:20: [ 246.785970] INFO: task kworker/u32:3:110 blocked for more than 120 seconds. 2022-05-27 04:19:20: [ 246.787022] Tainted: G X -------- Version-Release number of selected component (if applicable): Red Hat Enterprise Linux release 9.1 Beta (Plow) 5.14.0-96.el9.x86_64 qemu-kvm-7.0.0-4.el9.x86_64 seabios-bin-1.16.0-2.el9.noarch edk2-ovmf-20220221gitb24306f15d-1.el9.noarch virtio-win-prewhql-0.1-219.iso Guest:rhel8.6 or rhel9.1 How reproducible: 100% Steps to Reproduce: 1.boot vm [ -f /home/kvm_autotest_root/images/OVMF_VARS_x.fd ] || { cp -f /usr/share/OVMF/OVMF_VARS.fd /home/kvm_autotest_root/images/OVMF_VARS_x.fd; } /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \ -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kvm_autotest_root/images/OVMF_VARS_x.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \ -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 12288 \ -object memory-backend-ram,size=12288M,id=mem-machine_mem \ -smp 16 \ -cpu 'EPYC-Rome',+kvm_pv_unhalt \ \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -object iothread,id=iothread0 \ -object iothread,id=iothread1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/rhel910-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-3,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg0,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg0.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg0,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg0 \ -device scsi-hd,scsi-id=0,lun=0,id=stg0,bus=virtio_scsi_pci1.0,drive=drive_stg0,write-cache=on \ -blockdev node-name=file_stg1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg1.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg1 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg1,bus=virtio_scsi_pci1.0,drive=drive_stg1,write-cache=on \ -blockdev node-name=file_stg2,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg2.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg2,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg2 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg2,bus=virtio_scsi_pci1.0,drive=drive_stg2,write-cache=on \ -blockdev node-name=file_stg3,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg3.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg3,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg3 \ -device scsi-hd,scsi-id=127,lun=0,id=stg3,bus=virtio_scsi_pci1.0,drive=drive_stg3,write-cache=on \ -blockdev node-name=file_stg4,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg4.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg4,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg4 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg4,bus=virtio_scsi_pci1.0,drive=drive_stg4,write-cache=on \ -blockdev node-name=file_stg5,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg5.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg5,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg5 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg5,bus=virtio_scsi_pci1.0,drive=drive_stg5,write-cache=on \ -blockdev node-name=file_stg6,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg6.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg6,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg6 \ -device scsi-hd,scsi-id=254,lun=0,id=stg6,bus=virtio_scsi_pci1.0,drive=drive_stg6,write-cache=on \ -blockdev node-name=file_stg7,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg7.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg7,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg7 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg7,bus=virtio_scsi_pci1.0,drive=drive_stg7,write-cache=on \ -blockdev node-name=file_stg8,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg8.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg8,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg8 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg8,bus=virtio_scsi_pci1.0,drive=drive_stg8,write-cache=on \ -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \ -device virtio-scsi-pci,id=virtio_scsi_pci2,bus=pcie-root-port-4,addr=0x0,iothread=iothread0 \ -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \ -device virtio-scsi-pci,id=virtio_scsi_pci3,bus=pcie-root-port-5,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_stg9,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg9.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg9,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg9 \ -device scsi-hd,scsi-id=0,lun=0,id=stg9,bus=virtio_scsi_pci3.0,drive=drive_stg9,write-cache=on \ -blockdev node-name=file_stg10,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg10.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg10,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg10 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg10,bus=virtio_scsi_pci3.0,drive=drive_stg10,write-cache=on \ -blockdev node-name=file_stg11,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg11.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg11,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg11 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg11,bus=virtio_scsi_pci3.0,drive=drive_stg11,write-cache=on \ -blockdev node-name=file_stg12,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg12.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg12,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg12 \ -device scsi-hd,scsi-id=127,lun=0,id=stg12,bus=virtio_scsi_pci3.0,drive=drive_stg12,write-cache=on \ -blockdev node-name=file_stg13,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg13.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg13,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg13 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg13,bus=virtio_scsi_pci3.0,drive=drive_stg13,write-cache=on \ -blockdev node-name=file_stg14,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg14.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg14,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg14 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg14,bus=virtio_scsi_pci3.0,drive=drive_stg14,write-cache=on \ -blockdev node-name=file_stg15,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg15.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg15,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg15 \ -device scsi-hd,scsi-id=254,lun=0,id=stg15,bus=virtio_scsi_pci3.0,drive=drive_stg15,write-cache=on \ -blockdev node-name=file_stg16,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg16.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg16,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg16 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg16,bus=virtio_scsi_pci3.0,drive=drive_stg16,write-cache=on \ -blockdev node-name=file_stg17,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg17.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg17,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg17 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg17,bus=virtio_scsi_pci3.0,drive=drive_stg17,write-cache=on \ -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x1.0x6,bus=pcie.0,chassis=7 \ -device virtio-scsi-pci,id=virtio_scsi_pci4,bus=pcie-root-port-6,addr=0x0,iothread=iothread1 \ -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x1.0x7,bus=pcie.0,chassis=8 \ -device virtio-scsi-pci,id=virtio_scsi_pci5,bus=pcie-root-port-7,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg18,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg18.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg18,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg18 \ -device scsi-hd,scsi-id=0,lun=0,id=stg18,bus=virtio_scsi_pci5.0,drive=drive_stg18,write-cache=on \ -blockdev node-name=file_stg19,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg19.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg19,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg19 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg19,bus=virtio_scsi_pci5.0,drive=drive_stg19,write-cache=on \ -blockdev node-name=file_stg20,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg20.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg20,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg20 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg20,bus=virtio_scsi_pci5.0,drive=drive_stg20,write-cache=on \ -blockdev node-name=file_stg21,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg21.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg21,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg21 \ -device scsi-hd,scsi-id=127,lun=0,id=stg21,bus=virtio_scsi_pci5.0,drive=drive_stg21,write-cache=on \ -blockdev node-name=file_stg22,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg22.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg22,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg22 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg22,bus=virtio_scsi_pci5.0,drive=drive_stg22,write-cache=on \ -blockdev node-name=file_stg23,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg23.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg23,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg23 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg23,bus=virtio_scsi_pci5.0,drive=drive_stg23,write-cache=on \ -blockdev node-name=file_stg24,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg24.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg24,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg24 \ -device scsi-hd,scsi-id=254,lun=0,id=stg24,bus=virtio_scsi_pci5.0,drive=drive_stg24,write-cache=on \ -blockdev node-name=file_stg25,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg25.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg25,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg25 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg25,bus=virtio_scsi_pci5.0,drive=drive_stg25,write-cache=on \ -blockdev node-name=file_stg26,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg26.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg26,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg26 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg26,bus=virtio_scsi_pci5.0,drive=drive_stg26,write-cache=on \ -device pcie-root-port,id=pcie-root-port-8,port=0x8,multifunction=on,bus=pcie.0,addr=0x3,chassis=9 \ -device virtio-scsi-pci,id=virtio_scsi_pci6,bus=pcie-root-port-8,addr=0x0,iothread=iothread0 \ -device pcie-root-port,id=pcie-root-port-9,port=0x9,addr=0x3.0x1,bus=pcie.0,chassis=10 \ -device virtio-scsi-pci,id=virtio_scsi_pci7,bus=pcie-root-port-9,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_stg27,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg27.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg27,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg27 \ -device scsi-hd,scsi-id=0,lun=0,id=stg27,bus=virtio_scsi_pci7.0,drive=drive_stg27,write-cache=on \ -blockdev node-name=file_stg28,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg28.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg28,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg28 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg28,bus=virtio_scsi_pci7.0,drive=drive_stg28,write-cache=on \ -blockdev node-name=file_stg29,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg29.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg29,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg29 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg29,bus=virtio_scsi_pci7.0,drive=drive_stg29,write-cache=on \ -blockdev node-name=file_stg30,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg30.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg30,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg30 \ -device scsi-hd,scsi-id=127,lun=0,id=stg30,bus=virtio_scsi_pci7.0,drive=drive_stg30,write-cache=on \ -blockdev node-name=file_stg31,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg31.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg31,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg31 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg31,bus=virtio_scsi_pci7.0,drive=drive_stg31,write-cache=on \ -blockdev node-name=file_stg32,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg32.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg32,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg32 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg32,bus=virtio_scsi_pci7.0,drive=drive_stg32,write-cache=on \ -blockdev node-name=file_stg33,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg33.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg33,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg33 \ -device scsi-hd,scsi-id=254,lun=0,id=stg33,bus=virtio_scsi_pci7.0,drive=drive_stg33,write-cache=on \ -blockdev node-name=file_stg34,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg34.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg34,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg34 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg34,bus=virtio_scsi_pci7.0,drive=drive_stg34,write-cache=on \ -blockdev node-name=file_stg35,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg35.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg35,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg35 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg35,bus=virtio_scsi_pci7.0,drive=drive_stg35,write-cache=on \ -device pcie-root-port,id=pcie-root-port-10,port=0xa,addr=0x3.0x2,bus=pcie.0,chassis=11 \ -device virtio-scsi-pci,id=virtio_scsi_pci8,bus=pcie-root-port-10,addr=0x0,iothread=iothread1 \ -device pcie-root-port,id=pcie-root-port-11,port=0xb,addr=0x3.0x3,bus=pcie.0,chassis=12 \ -device virtio-scsi-pci,id=virtio_scsi_pci9,bus=pcie-root-port-11,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg36,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg36.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg36,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg36 \ -device scsi-hd,scsi-id=0,lun=0,id=stg36,bus=virtio_scsi_pci9.0,drive=drive_stg36,write-cache=on \ -blockdev node-name=file_stg37,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg37.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg37,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg37 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg37,bus=virtio_scsi_pci9.0,drive=drive_stg37,write-cache=on \ -blockdev node-name=file_stg38,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg38.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg38,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg38 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg38,bus=virtio_scsi_pci9.0,drive=drive_stg38,write-cache=on \ -blockdev node-name=file_stg39,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg39.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg39,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg39 \ -device scsi-hd,scsi-id=127,lun=0,id=stg39,bus=virtio_scsi_pci9.0,drive=drive_stg39,write-cache=on \ -blockdev node-name=file_stg40,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg40.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg40,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg40 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg40,bus=virtio_scsi_pci9.0,drive=drive_stg40,write-cache=on \ -blockdev node-name=file_stg41,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg41.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg41,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg41 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg41,bus=virtio_scsi_pci9.0,drive=drive_stg41,write-cache=on \ -blockdev node-name=file_stg42,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg42.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg42,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg42 \ -device scsi-hd,scsi-id=254,lun=0,id=stg42,bus=virtio_scsi_pci9.0,drive=drive_stg42,write-cache=on \ -blockdev node-name=file_stg43,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg43.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg43,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg43 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg43,bus=virtio_scsi_pci9.0,drive=drive_stg43,write-cache=on \ -blockdev node-name=file_stg44,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg44.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg44,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg44 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg44,bus=virtio_scsi_pci9.0,drive=drive_stg44,write-cache=on \ -device pcie-root-port,id=pcie-root-port-12,port=0xc,addr=0x3.0x4,bus=pcie.0,chassis=13 \ -device virtio-scsi-pci,id=virtio_scsi_pci10,bus=pcie-root-port-12,addr=0x0,iothread=iothread0 \ -device pcie-root-port,id=pcie-root-port-13,port=0xd,addr=0x3.0x5,bus=pcie.0,chassis=14 \ -device virtio-scsi-pci,id=virtio_scsi_pci11,bus=pcie-root-port-13,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_stg45,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg45.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg45,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg45 \ -device scsi-hd,scsi-id=0,lun=0,id=stg45,bus=virtio_scsi_pci11.0,drive=drive_stg45,write-cache=on \ -blockdev node-name=file_stg46,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg46.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg46,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg46 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg46,bus=virtio_scsi_pci11.0,drive=drive_stg46,write-cache=on \ -blockdev node-name=file_stg47,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg47.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg47,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg47 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg47,bus=virtio_scsi_pci11.0,drive=drive_stg47,write-cache=on \ -blockdev node-name=file_stg48,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg48.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg48,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg48 \ -device scsi-hd,scsi-id=127,lun=0,id=stg48,bus=virtio_scsi_pci11.0,drive=drive_stg48,write-cache=on \ -blockdev node-name=file_stg49,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg49.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg49,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg49 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg49,bus=virtio_scsi_pci11.0,drive=drive_stg49,write-cache=on \ -blockdev node-name=file_stg50,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg50.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg50,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg50 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg50,bus=virtio_scsi_pci11.0,drive=drive_stg50,write-cache=on \ -blockdev node-name=file_stg51,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg51.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg51,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg51 \ -device scsi-hd,scsi-id=254,lun=0,id=stg51,bus=virtio_scsi_pci11.0,drive=drive_stg51,write-cache=on \ -blockdev node-name=file_stg52,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg52.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg52,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg52 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg52,bus=virtio_scsi_pci11.0,drive=drive_stg52,write-cache=on \ -blockdev node-name=file_stg53,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg53.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg53,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg53 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg53,bus=virtio_scsi_pci11.0,drive=drive_stg53,write-cache=on \ -device pcie-root-port,id=pcie-root-port-14,port=0xe,addr=0x3.0x6,bus=pcie.0,chassis=15 \ -device virtio-scsi-pci,id=virtio_scsi_pci12,bus=pcie-root-port-14,addr=0x0,iothread=iothread1 \ -device pcie-root-port,id=pcie-root-port-15,port=0xf,addr=0x3.0x7,bus=pcie.0,chassis=16 \ -device virtio-scsi-pci,id=virtio_scsi_pci13,bus=pcie-root-port-15,addr=0x0,iothread=iothread1 \ -blockdev node-name=file_stg54,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg54.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg54,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg54 \ -device scsi-hd,scsi-id=0,lun=0,id=stg54,bus=virtio_scsi_pci13.0,drive=drive_stg54,write-cache=on \ -blockdev node-name=file_stg55,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg55.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg55,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg55 \ -device scsi-hd,scsi-id=0,lun=8191,id=stg55,bus=virtio_scsi_pci13.0,drive=drive_stg55,write-cache=on \ -blockdev node-name=file_stg56,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg56.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg56,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg56 \ -device scsi-hd,scsi-id=0,lun=16382,id=stg56,bus=virtio_scsi_pci13.0,drive=drive_stg56,write-cache=on \ -blockdev node-name=file_stg57,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg57.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg57,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg57 \ -device scsi-hd,scsi-id=127,lun=0,id=stg57,bus=virtio_scsi_pci13.0,drive=drive_stg57,write-cache=on \ -blockdev node-name=file_stg58,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg58.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg58,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg58 \ -device scsi-hd,scsi-id=127,lun=8191,id=stg58,bus=virtio_scsi_pci13.0,drive=drive_stg58,write-cache=on \ -blockdev node-name=file_stg59,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg59.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg59,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg59 \ -device scsi-hd,scsi-id=127,lun=16382,id=stg59,bus=virtio_scsi_pci13.0,drive=drive_stg59,write-cache=on \ -blockdev node-name=file_stg60,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg60.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg60,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg60 \ -device scsi-hd,scsi-id=254,lun=0,id=stg60,bus=virtio_scsi_pci13.0,drive=drive_stg60,write-cache=on \ -blockdev node-name=file_stg61,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg61.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg61,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg61 \ -device scsi-hd,scsi-id=254,lun=8191,id=stg61,bus=virtio_scsi_pci13.0,drive=drive_stg61,write-cache=on \ -blockdev node-name=file_stg62,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/mstg62.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_stg62,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg62 \ -device scsi-hd,scsi-id=254,lun=16382,id=stg62,bus=virtio_scsi_pci13.0,drive=drive_stg62,write-cache=on \ -device pcie-root-port,id=pcie-root-port-16,port=0x10,multifunction=on,bus=pcie.0,addr=0x4,chassis=17 \ -device virtio-net-pci,mac=9a:33:ae:2d:a9:17,id=id6Gkz3l,netdev=idK0Huuo,bus=pcie-root-port-16,addr=0x0 \ -netdev tap,id=idK0Huuo,vhost=on \ -vnc :5 \ -qmp tcp:0:5955,server=on,wait=off \ -monitor stdio \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x5,chassis=18 2. 3. Actual results: boot hang and kernel crash Expected results: boot succeed Additional info: cpu info Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 43 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD BIOS Vendor ID: AMD Model name: AMD EPYC 7302P 16-Core Processor BIOS Model name: AMD EPYC 7302P 16-Core Processor CPU family: 23 Model: 49 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 0 BogoMIPS: 5989.05 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr ss e sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_t sc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2a pic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm ss e4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase b mi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wb noinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeass ists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_reco v succor smca sme sev sev_es 1.boot with seabios no issue found 2.if remove smp or change smp to 2 , it may pass 3.Same test pass on other qemu/ovmf but the cpu is not same (intel) Red Hat Enterprise Linux release 9.0 (Plow) 5.14.0-70.13.1.el9_0.x86_64 qemu-kvm-6.2.0-11.el9_0.2.x86_64 seabios-bin-1.15.0-1.el9.noarch edk2-ovmf-20220126gitbb1bba3d77-3.el9.noarch virtio-win-prewhql-0.1-219.iso Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Vendor ID: GenuineIntel BIOS Vendor ID: Intel Model name: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz BIOS Model name: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz CPU family: 6 Model: 85 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 1 Stepping: 7 CPU max MHz: 3200.0000 CPU min MHz: 1000.0000 BogoMIPS: 4400.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mm x fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon peb s bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_ cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcn t tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ca t_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnm i flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm id a arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities