Bug 1671902

This is a SeaBIOS bug, it is possible to create QEMU configurations that exceed the available I/O port space.  Failure to assign I/O port space to a PCI Express hierarchy should not be considered a fatal error.  The guest would likely work fine if SeaBIOS had left the bridge I/O port apertures unprogrammed.

- This BZ seems to be a duplicate (or "reincarnation") of
  <https://bugzilla.redhat.com/show_bug.cgi?id=1271457>, from 2015/2016.

- The BZ specifies "e1000e", that is, a PCI Express device, which is
  supposed to work fine without IO BARs. Therefore what I think is
  missing is the "io-reserve=0" property from the domain configuration
  (for the "pcie-root-port" devices), which we introduced for
  <https://bugzilla.redhat.com/show_bug.cgi?id=1344299>.

reproduce this bug with fast train
 
host version:
qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64
kernel-4.18.0-137.el8.x86_64
seabios-1.12.0-5.module+el8.1.0+4022+29a53beb.x86_64
guest:rhel8.1.0

Hit this exist bug:

Host version:
seabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch
kernel-4.18.0-147.8.el8.x86_64
qemu-kvm-2.12.0-89.module+el8.2.0+4436+f3a2188d.x86_64
host:rhel8.2.0
guest:rhel8.2.0
If this is a new problem,please tell me know,I will reopen a new BZ,thanks a lot.

Hit this bug with rhel7.8 guest in rhel7.8 host

host version:
qemu-kvm-rhev-2.12.0-38.el7.x86_64
kernel-3.10.0-1112.el7.x86_64
seabios-1.11.0-2.el7.x86_64q


Hello,

The RHEL7.8 product has the same problem. Do I need to clone this bug into the RHEL7.8 product ?


thanks
yiqian

> - The BZ specifies "e1000e", that is, a PCI Express device, which is
>   supposed to work fine without IO BARs. Therefore what I think is
>   missing is the "io-reserve=0" property from the domain configuration
>   (for the "pcie-root-port" devices), which we introduced for
>   <https://bugzilla.redhat.com/show_bug.cgi?id=1344299>.

This.

clearing needinfo (bug has been closed)

Hi, Gerd

I have some question confirm with you,could you help me check the following, please:
1. According to bug 1344299 content, if we add parameters "io-reserve=0" at pcie-root-port device,so it can supports more than 15 e1000e devices?
2. Is the address space occupied by the two devices virtio-net and e1000e different? e1000e device occupy more address space?
3. According to bug 1344299 content, I tried test it with matrix "seabios+q35+28 e1000e devices" and add parameters "io-reserver=0" at pcie-root-port devices. But the guest can not boot up, its VNC info show as "Guest has not initialized the display". However, it can start normally when only 16 e1000e devices are added. Is this normally?

Test Version:
qemu-kvm-4.2.0-35.module+el8.4.0+8705+34397d87.x86_64
kernel-4.18.0-246.el8.dt2.x86_64
seabios-bin-1.13.0-2.module+el8.3.0+7353+9de0a3cc.noarch
Guest:
rhel8.4 guest and windows 2019

qemu cli:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-machine q35 \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x2 \
-m 7168  \
-smp 6,maxcpus=6,cores=3,threads=1,dies=1,sockets=2  \
-cpu 'Haswell-noTSX',+kvm_pv_unhalt \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
-blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel840-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:45,id=idXmzEcq,netdev=idzfPNvd,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idzfPNvd \
-device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:46,id=idkw9I7G,netdev=idwsC8KD,bus=pcie-root-port-4,addr=0x0  \
-netdev tap,id=idwsC8KD \
-device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:47,id=idi7k0cI,netdev=idd18XV2,bus=pcie-root-port-5,addr=0x0  \
-netdev tap,id=idd18XV2 \
-device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x1.0x6,bus=pcie.0,chassis=7,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:48,id=idhHyRK6,netdev=idl5gFTD,bus=pcie-root-port-6,addr=0x0  \
-netdev tap,id=idl5gFTD \
-device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x1.0x7,bus=pcie.0,chassis=8,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:49,id=idOpZLG5,netdev=idQhuoM1,bus=pcie-root-port-7,addr=0x0  \
-netdev tap,id=idQhuoM1 \
-device pcie-root-port,id=pcie-root-port-8,port=0x8,multifunction=on,bus=pcie.0,addr=0x3,chassis=9,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:4a,id=id5M7EpG,netdev=idRAnv0c,bus=pcie-root-port-8,addr=0x0  \
-netdev tap,id=idRAnv0c \
-device pcie-root-port,id=pcie-root-port-9,port=0x9,addr=0x3.0x1,bus=pcie.0,chassis=10,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:4b,id=ide1nIjf,netdev=id16dIlz,bus=pcie-root-port-9,addr=0x0  \
-netdev tap,id=id16dIlz \
-device pcie-root-port,id=pcie-root-port-10,port=0xa,addr=0x3.0x2,bus=pcie.0,chassis=11,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:4c,id=idBp1Ntc,netdev=idpnV7kT,bus=pcie-root-port-10,addr=0x0  \
-netdev tap,id=idpnV7kT \
-device pcie-root-port,id=pcie-root-port-11,port=0xb,addr=0x3.0x3,bus=pcie.0,chassis=12,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:4d,id=idwIkA0q,netdev=idWrpN7W,bus=pcie-root-port-11,addr=0x0  \
-netdev tap,id=idWrpN7W \
-device pcie-root-port,id=pcie-root-port-12,port=0xc,addr=0x3.0x4,bus=pcie.0,chassis=13,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:4e,id=idjzElXz,netdev=idJWJ0IN,bus=pcie-root-port-12,addr=0x0  \
-netdev tap,id=idJWJ0IN \
-device pcie-root-port,id=pcie-root-port-13,port=0xd,addr=0x3.0x5,bus=pcie.0,chassis=14,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:4f,id=idcpJsrR,netdev=idfcHErX,bus=pcie-root-port-13,addr=0x0  \
-netdev tap,id=idfcHErX \
-device pcie-root-port,id=pcie-root-port-14,port=0xe,addr=0x3.0x6,bus=pcie.0,chassis=15,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:50,id=idbhX8Mv,netdev=idw9eqiJ,bus=pcie-root-port-14,addr=0x0  \
-netdev tap,id=idw9eqiJ \
-device pcie-root-port,id=pcie-root-port-15,port=0xf,addr=0x3.0x7,bus=pcie.0,chassis=16,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:51,id=id82GRCH,netdev=idzdtvLg,bus=pcie-root-port-15,addr=0x0  \
-netdev tap,id=idzdtvLg \
-device pcie-root-port,id=pcie-root-port-17,port=0x11,addr=0x4.0x1,bus=pcie.0,chassis=18,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:53,id=idAsknkA,netdev=idogxfSa,bus=pcie-root-port-17,addr=0x0  \
-netdev tap,id=idogxfSa \
-device pcie-root-port,id=pcie-root-port-18,port=0x12,addr=0x4.0x2,bus=pcie.0,chassis=19,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:54,id=idnq9mlX,netdev=idtcCaVC,bus=pcie-root-port-18,addr=0x0  \
-netdev tap,id=idtcCaVC \
-device pcie-root-port,id=pcie-root-port-19,port=0x13,addr=0x4.0x3,bus=pcie.0,chassis=20,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:55,id=id4qvXLs,netdev=idAqG5Ze,bus=pcie-root-port-19,addr=0x0  \
-netdev tap,id=idAqG5Ze \
-device pcie-root-port,id=pcie-root-port-20,port=0x14,addr=0x4.0x4,bus=pcie.0,chassis=21,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:56,id=idPDl75Z,netdev=idfhz7R4,bus=pcie-root-port-20,addr=0x0  \
-netdev tap,id=idfhz7R4 \
-device pcie-root-port,id=pcie-root-port-21,port=0x15,addr=0x4.0x5,bus=pcie.0,chassis=22,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:57,id=idVlGlay,netdev=idI1NO8t,bus=pcie-root-port-21,addr=0x0  \
-netdev tap,id=idI1NO8t \
-device pcie-root-port,id=pcie-root-port-22,port=0x16,addr=0x4.0x6,bus=pcie.0,chassis=23,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:58,id=ido09oDj,netdev=idyryIi0,bus=pcie-root-port-22,addr=0x0  \
-netdev tap,id=idyryIi0 \
-device pcie-root-port,id=pcie-root-port-23,port=0x17,addr=0x4.0x7,bus=pcie.0,chassis=24,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:59,id=idksAi7u,netdev=idRh76Av,bus=pcie-root-port-23,addr=0x0  \
-netdev tap,id=idRh76Av \
-device pcie-root-port,id=pcie-root-port-24,port=0x18,multifunction=on,bus=pcie.0,addr=0x5,chassis=25,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:5a,id=idlM1Zn7,netdev=id7xypn2,bus=pcie-root-port-24,addr=0x0  \
-netdev tap,id=id7xypn2 \
-device pcie-root-port,id=pcie-root-port-25,port=0x19,addr=0x5.0x1,bus=pcie.0,chassis=26,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:5b,id=idkqzhjV,netdev=idtpwN9H,bus=pcie-root-port-25,addr=0x0  \
-netdev tap,id=idtpwN9H \
-device pcie-root-port,id=pcie-root-port-26,port=0x1a,addr=0x5.0x2,bus=pcie.0,chassis=27,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:5c,id=idS8gqWy,netdev=id4LPAFV,bus=pcie-root-port-26,addr=0x0  \
-netdev tap,id=id4LPAFV \
-device pcie-root-port,id=pcie-root-port-27,port=0x1b,addr=0x5.0x3,bus=pcie.0,chassis=28,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:5d,id=idhEcU3N,netdev=idyH544z,bus=pcie-root-port-27,addr=0x0  \
-netdev tap,id=idyH544z \
-device pcie-root-port,id=pcie-root-port-28,port=0x1c,addr=0x5.0x4,bus=pcie.0,chassis=29,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:5e,id=idMrj80J,netdev=idJlVA65,bus=pcie-root-port-28,addr=0x0  \
-netdev tap,id=idJlVA65 \
-device pcie-root-port,id=pcie-root-port-29,port=0x1d,addr=0x5.0x5,bus=pcie.0,chassis=30,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:5f,id=id8wxDFC,netdev=idGfz9oV,bus=pcie-root-port-29,addr=0x0  \
-netdev tap,id=idGfz9oV \
-vnc :0  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x6,chassis=31 \
-monitor stdio \

If my test command line is error, please let me know. Thanks in advance.

Best Regards
Lei Yang

> 1. According to bug 1344299 content, if we add parameters "io-reserve=0" at
> pcie-root-port device,so it can supports more than 15 e1000e devices?

In theory yes.  According to the PCIe spec devices are required to work fine
even without ioports, using mmio instead.

> 2. Is the address space occupied by the two devices virtio-net and e1000e
> different? e1000e device occupy more address space?

virtio-net has no ioports, it uses mmio exclusively.
seabios doesn't assign an ioport window to the pcie root port in that case,
so it works without explicit io-reserve=0.

> 3. According to bug 1344299 content, I tried test it with matrix
> "seabios+q35+28 e1000e devices" and add parameters "io-reserver=0" at
> pcie-root-port devices. But the guest can not boot up, its VNC info show as
> "Guest has not initialized the display". However, it can start normally when
> only 16 e1000e devices are added. Is this normally?

Possibly seabios runs out of memory.

You can try set the romfile property for the nics to the empty string,
so seabios wouldn't load 28 pxe option roms ...

> -device
> pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,
> chassis=1 \

> -device VGA,bus=pcie.0,addr=0x2 \

I'm not sure addr=2 is allowed for devices in a pcie root port.
Try addr=0 instead (or drop it).

When plugging the display device into a pcie root port I'd recommend
using -device bochs-display which is fully functional as PCIe device.
VGA may not work, or only offer limited functionality.

(In reply to Gerd Hoffmann from comment #12)

Hi, Gerd

First of all, thank you very much for your reply. The information you provide is very important for my testing.
Then I did some tests based on the information you provided, the following are test results:

=>Test combination 1 -> (VGA + Seabios + e1000e device) -> It can support 14 e1000e devices
-device VGA,bus=pcie.0,addr=0x2 \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device e1000e,mac=9a:82:a8:f7:3c:45,id=idXmzEcq,netdev=idzfPNvd,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idzfPNvd \

=>Test combination 2 -> (VGA + Seabios + e1000e device + io-reserve=0) -> it can support 16 e1000e devices
-device VGA,bus=pcie.0,addr=0x2 \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:45,id=idXmzEcq,netdev=idzfPNvd,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idzfPNvd \

=>Test combination 3 -> (-device bochs-display + Seabios + e1000e device + io-reserve=0 + romfile='') -> it can support 22 e1000e devices
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2,io-reserve=0 \
-device bochs-display,bus=pcie-root-port-1,addr=0x0 \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4,io-reserve=0 \
-device e1000e,mac=9a:82:a8:f7:3c:45,id=idXmzEcq,netdev=idzfPNvd,bus=pcie-root-port-3,addr=0x0,romfile=''  \
-netdev tap,id=idzfPNvd \

Based on above test results, I have some question confirm with you and look forward to your reply,please:
1. As you said, it should support more devices in theory, but currently there are only 22 devices at most. The reason for this phenomenon is because the support of parameters 'io-reserve=0' is not friendly enough? 
2. If we do not provide more support, in future tests, should I choose which combination of tests is more reasonable?
3. What is the reason we closed this bug? Is it because the e1000e devices is being used less and less by customers, so we don't need provide more support for it in the guest? If it is like what I said, is it more appropriate to change the status of this bug to "WONTFIX" ?

Best Regards
Lei Yang

> =>Test combination 3 -> (-device bochs-display + Seabios + e1000e device +
> io-reserve=0 + romfile='') -> it can support 22 e1000e devices

> Based on above test results, I have some question confirm with you and look
> forward to your reply,please:
> 1. As you said, it should support more devices in theory, but currently
> there are only 22 devices at most. The reason for this phenomenon is because
> the support of parameters 'io-reserve=0' is not friendly enough? 

Well, seabios runs in real mode so we have to live with limited resources,
especially memory.  Which in turn limits the amount of devices seabios can
support.  In most case I've seen so far it is a memory allocation failure
when seabios hits a limit.  Where exactly the limit is can vary depending
on device and seabios version.  And there isn't much we can do about it,
other than switching to a more modern firmware (ovmf) which hasn't its roots
in the 80ies of the last century.

I've expected test #3 can deal with more than 22 devices.  I can have a look
if you attach the seabios log.  Most likely it is a memory allocation failure.
Should that not be the case we maybe can do something about it.

> 2. If we do not provide more support, in future tests, should I choose which
> combination of tests is more reasonable?

I'd suggest to simply limit the number of e1000e devices to something like 8.
I know we have documented limits for storage devices (sata, virtio-blk,
virtio-scsi) somewhere.  We can do the same for nics should that not be the
case already.

> 3. What is the reason we closed this bug? Is it because the e1000e devices
> is being used less and less by customers, so we don't need provide more
> support for it in the guest?

See above.  Most likely we can't do much about it.

Also e1000e is not the best choice performance-wise, better use virtio-net,
especially if you want run a guest with *alot* of nics.  So while e1000e
clearly has its use cases, especially for guest which don't ship with virtio
drivers, I consider the combination of e1000e + lots of nics not that important.

> If it is like what I said, is it more
> appropriate to change the status of this bug to "WONTFIX" ?

No.  Maybe CANTFIX.

(In reply to Gerd Hoffmann from comment #14)
> > =>Test combination 3 -> (-device bochs-display + Seabios + e1000e device +
> > io-reserve=0 + romfile='') -> it can support 22 e1000e devices
> 
> > Based on above test results, I have some question confirm with you and look
> > forward to your reply,please:
> > 1. As you said, it should support more devices in theory, but currently
> > there are only 22 devices at most. The reason for this phenomenon is because
> > the support of parameters 'io-reserve=0' is not friendly enough? 
> 
> Well, seabios runs in real mode so we have to live with limited resources,
> especially memory.  Which in turn limits the amount of devices seabios can
> support.  In most case I've seen so far it is a memory allocation failure
> when seabios hits a limit.  Where exactly the limit is can vary depending
> on device and seabios version.  And there isn't much we can do about it,
> other than switching to a more modern firmware (ovmf) which hasn't its roots
> in the 80ies of the last century.
> 
> I've expected test #3 can deal with more than 22 devices.  I can have a look
> if you attach the seabios log.  Most likely it is a memory allocation
> failure.
> Should that not be the case we maybe can do something about it.
I checked the seabios log carefully and it was the same as the bug description, so it was still a memory allocation problem.
> 
> > 2. If we do not provide more support, in future tests, should I choose which
> > combination of tests is more reasonable?
> 
> I'd suggest to simply limit the number of e1000e devices to something like 8.
> I know we have documented limits for storage devices (sata, virtio-blk,
> virtio-scsi) somewhere.  We can do the same for nics should that not be the
> case already.
Thanks, I will change my test case based on this information.
> 
> > 3. What is the reason we closed this bug? Is it because the e1000e devices
> > is being used less and less by customers, so we don't need provide more
> > support for it in the guest?
> 
> See above.  Most likely we can't do much about it.
> 
> Also e1000e is not the best choice performance-wise, better use virtio-net,
> especially if you want run a guest with *alot* of nics.  So while e1000e
> clearly has its use cases, especially for guest which don't ship with virtio
> drivers, I consider the combination of e1000e + lots of nics not that
> important.
> 
> > If it is like what I said, is it more
> > appropriate to change the status of this bug to "WONTFIX" ?
> 
> No.  Maybe CANTFIX.
Based on above, change the status to "CANTFIX".