Hide Forgot
Description of problem: Out of I/O address space when boot up with 15 virtio-blk-pci data disks via PCIe-switch, and guest didn't boot up Version-Release number of selected component (if applicable): kernel-3.10.0-373.el7.x86_64 qemu-img-rhev-2.5.0-4.el7.x86_64 seabios-1.9.1-2.el7.x86_64 How reproducible: 3/3 Steps to Reproduce: 1.boot guest as the following via multi-pcie.sh shell script. # ./multi-pcie.sh 15 # cat multi-pcie.sh #!/bin/sh MACHINE=q35 SMP=4,cores=2,threads=2,sockets=1 MEM=2G GUEST_IMG=/home/rhel.img IMG_FORMAT=raw CLI="gdb --args /usr/libexec/qemu-kvm -enable-kvm -M $MACHINE -smp $SMP -m $MEM -name vm1 -drive file=$GUEST_IMG,if=none,id=guest-img,format=$IMG_FORMAT,werror=stop,rerror=stop -device ide-hd,drive=guest-img,bus=ide.0,unit=0,id=os-disk,bootindex=0 -spice port=5931,disable-ticketing -vga qxl -monitor stdio -serial unix:/tmp/console,server,nowait -qmp tcp:0:6666,server,nowait -chardev file,path=/home/seabios.log,id=seabios -device isa-debugcon,chardev=seabios,iobase=0x402 -boot menu=on,reboot-timeout=8,strict=on " echo before juli while [ ${i:=0} -lt ${1:-0} ] do dstreamId=$((i)) ustreamId=$((dstreamId/32)) chassisId=$((dstreamId+1)) blkDiskId=$((dstreamId)) if [ $((dstreamId%32)) -eq 0 ] then CLI="$CLI -device ioh3420,bus=pcie.0,id=root.$ustreamId,slot=$ustreamId" CLI="$CLI -device x3130-upstream,bus=root.$ustreamId,id=upstream$ustreamId" fi qemu-img create -f qcow2 /home/disk/disk$blkDiskId 100M echo "juli$i" CLI="$CLI -device xio3130-downstream,bus=upstream$ustreamId,id=downstream$dstreamId,chassis=$chassisId" CLI="$CLI -drive file=/home/disk/disk$blkDiskId,if=none,id=disk$blkDiskId,format=qcow2 " CLI="$CLI -device virtio-blk-pci,scsi=off,drive=disk$blkDiskId,id=virtio-blk$blkDiskId,bus=downstream$dstreamId" i=$((i+1)) done $CLI Actual results: guest didn't boot up and using isa-debugcon debug, got the following messages: [root@localhost home]# cat seabios.log SeaBIOS (version 1.9.1-2.el7) BUILD: gcc: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) binutils: version 2.23.52.0.1-55.el7 20130226 Running on QEMU (q35) Running on KVM RamSize: 0x80000000 [cmos] Relocating init from 0x000e08b0 to 0x7ffb09e0 (size 62864) Found QEMU fw_cfg QEMU fw_cfg DMA interface supported RamBlock: addr 0x0000000000000000 len 0x0000000080000000 [e820] Moving pm_base to 0x600 boot order: 1: /pci@i0cf8/pci8086,2922@1f,2/drive@0/disk@0 2: HALT === PCI bus & bridge init === PCI: pci_bios_init_bus_rec bus = 0x0 PCI: pci_bios_init_bus_rec bdf = 0x18 PCI: primary bus = 0x0 PCI: secondary bus = 0xff -> 0x1 PCI: pci_bios_init_bus_rec bus = 0x1 PCI: pci_bios_init_bus_rec bdf = 0x100 PCI: primary bus = 0x0 -> 0x1 PCI: secondary bus = 0xff -> 0x2 PCI: pci_bios_init_bus_rec bus = 0x2 PCI: pci_bios_init_bus_rec bdf = 0x200 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x3 PCI: pci_bios_init_bus_rec bus = 0x3 PCI: subordinate bus = 0x0 -> 0x3 PCI: pci_bios_init_bus_rec bdf = 0x208 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x4 PCI: pci_bios_init_bus_rec bus = 0x4 PCI: subordinate bus = 0x0 -> 0x4 PCI: pci_bios_init_bus_rec bdf = 0x210 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x5 PCI: pci_bios_init_bus_rec bus = 0x5 PCI: subordinate bus = 0x0 -> 0x5 PCI: pci_bios_init_bus_rec bdf = 0x218 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x6 PCI: pci_bios_init_bus_rec bus = 0x6 PCI: subordinate bus = 0x0 -> 0x6 PCI: pci_bios_init_bus_rec bdf = 0x220 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x7 PCI: pci_bios_init_bus_rec bus = 0x7 PCI: subordinate bus = 0x0 -> 0x7 PCI: pci_bios_init_bus_rec bdf = 0x228 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x8 PCI: pci_bios_init_bus_rec bus = 0x8 PCI: subordinate bus = 0x0 -> 0x8 PCI: pci_bios_init_bus_rec bdf = 0x230 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x9 PCI: pci_bios_init_bus_rec bus = 0x9 PCI: subordinate bus = 0x0 -> 0x9 PCI: pci_bios_init_bus_rec bdf = 0x238 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0xa PCI: pci_bios_init_bus_rec bus = 0xa PCI: subordinate bus = 0x0 -> 0xa PCI: pci_bios_init_bus_rec bdf = 0x240 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0xb PCI: pci_bios_init_bus_rec bus = 0xb PCI: subordinate bus = 0x0 -> 0xb PCI: pci_bios_init_bus_rec bdf = 0x248 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0xc PCI: pci_bios_init_bus_rec bus = 0xc PCI: subordinate bus = 0x0 -> 0xc PCI: pci_bios_init_bus_rec bdf = 0x250 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0xd PCI: pci_bios_init_bus_rec bus = 0xd PCI: subordinate bus = 0x0 -> 0xd PCI: pci_bios_init_bus_rec bdf = 0x258 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0xe PCI: pci_bios_init_bus_rec bus = 0xe PCI: subordinate bus = 0x0 -> 0xe PCI: pci_bios_init_bus_rec bdf = 0x260 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0xf PCI: pci_bios_init_bus_rec bus = 0xf PCI: subordinate bus = 0x0 -> 0xf PCI: pci_bios_init_bus_rec bdf = 0x268 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x10 PCI: pci_bios_init_bus_rec bus = 0x10 PCI: subordinate bus = 0x0 -> 0x10 PCI: pci_bios_init_bus_rec bdf = 0x270 PCI: primary bus = 0x0 -> 0x2 PCI: secondary bus = 0xff -> 0x11 PCI: pci_bios_init_bus_rec bus = 0x11 PCI: subordinate bus = 0x0 -> 0x11 PCI: subordinate bus = 0x0 -> 0x11 PCI: subordinate bus = 0x0 -> 0x11 === PCI device probing === Found 38 PCI devices (max PCI bus is 11) === PCI new allocation pass #1 === PCI: check devices PCI: secondary bus 17 size 00001000 type io PCI: secondary bus 17 size 00200000 type mem PCI: secondary bus 17 size 00200000 type prefmem PCI: secondary bus 16 size 00001000 type io PCI: secondary bus 16 size 00200000 type mem PCI: secondary bus 16 size 00200000 type prefmem PCI: secondary bus 15 size 00001000 type io PCI: secondary bus 15 size 00200000 type mem PCI: secondary bus 15 size 00200000 type prefmem PCI: secondary bus 14 size 00001000 type io PCI: secondary bus 14 size 00200000 type mem PCI: secondary bus 14 size 00200000 type prefmem PCI: secondary bus 13 size 00001000 type io PCI: secondary bus 13 size 00200000 type mem PCI: secondary bus 13 size 00200000 type prefmem PCI: secondary bus 12 size 00001000 type io PCI: secondary bus 12 size 00200000 type mem PCI: secondary bus 12 size 00200000 type prefmem PCI: secondary bus 11 size 00001000 type io PCI: secondary bus 11 size 00200000 type mem PCI: secondary bus 11 size 00200000 type prefmem PCI: secondary bus 10 size 00001000 type io PCI: secondary bus 10 size 00200000 type mem PCI: secondary bus 10 size 00200000 type prefmem PCI: secondary bus 9 size 00001000 type io PCI: secondary bus 9 size 00200000 type mem PCI: secondary bus 9 size 00200000 type prefmem PCI: secondary bus 8 size 00001000 type io PCI: secondary bus 8 size 00200000 type mem PCI: secondary bus 8 size 00200000 type prefmem PCI: secondary bus 7 size 00001000 type io PCI: secondary bus 7 size 00200000 type mem PCI: secondary bus 7 size 00200000 type prefmem PCI: secondary bus 6 size 00001000 type io PCI: secondary bus 6 size 00200000 type mem PCI: secondary bus 6 size 00200000 type prefmem PCI: secondary bus 5 size 00001000 type io PCI: secondary bus 5 size 00200000 type mem PCI: secondary bus 5 size 00200000 type prefmem PCI: secondary bus 4 size 00001000 type io PCI: secondary bus 4 size 00200000 type mem PCI: secondary bus 4 size 00200000 type prefmem PCI: secondary bus 3 size 00001000 type io PCI: secondary bus 3 size 00200000 type mem PCI: secondary bus 3 size 00200000 type prefmem PCI: secondary bus 2 size 0000f000 type io PCI: secondary bus 2 size 01e00000 type mem PCI: secondary bus 2 size 01e00000 type prefmem PCI: secondary bus 1 size 0000f000 type io PCI: secondary bus 1 size 01e00000 type mem PCI: secondary bus 1 size 01e00000 type prefmem === PCI new allocation pass #2 === PCI: out of I/O address space Expected results: guest boot up and seabios didn't have error message Additional info: Hit the issue with following vesion: kernel-3.10.0-327.18.1.el7.x86_64.rpm qemu-img-rhev-2.3.0-31.el7_2.10.x86_64.rpm seabios-1.7.5-11.el7.x86_64.rpm
Hi, In PCIe ,each device sits behind its own pci bridge, and the PCI spec requires to reserve min 0x1000 IO space for each bridge (PCIe ports are PCI bridges). This is the maximum IO space the firmware can give. We already solved this by allowing virtio devices to be exposed as PCI Express devices and then we can "skip" reserving IO for the bridges with PCIe devices behind them. Please add -device virtio-blk-pci,disable-modern=off,disable-legacy=on,... and see if the problem is solved. Thanks, Marcel