RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1348798 - [Q35] No space for [io size 0x9000]
Summary: [Q35] No space for [io size 0x9000]
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ovmf
Version: 7.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Laszlo Ersek
QA Contact: FuXiangChun
URL:
Whiteboard:
Depends On: 1332408
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-22 06:27 UTC by jingzhao
Modified: 2017-01-27 14:11 UTC (History)
9 users (show)

Fixed In Version: ovmf-20160608-3.git988715a.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-27 14:11:05 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
dmesg log (81.72 KB, text/plain)
2016-06-22 06:27 UTC, jingzhao 		no flags Details
ovmf log (104.30 KB, text/plain)
2016-06-22 06:29 UTC, jingzhao 		no flags Details
View All

Description jingzhao 2016-06-22 06:27:20 UTC 
Description of problem:
dmesg error when boot guest with 1 upstream which attached with 9 downstreams

[    0.673583] pci 0000:00:04.0: BAR 13: no space for [io  size 0x9000]
[    0.673584] pci 0000:00:04.0: BAR 13: failed to assign [io  size 0x9000]


Version-Release number of selected component (if applicable):
kernel-3.10.0-447.el7.x86_64
qemu-img-rhev-2.6.0-7.el7.x86_64
OVMF-20160608-1.git988715a.el7.noarch

How reproducible:
3/3

Steps to Reproduce:
1. Boot guest with 1 upstream which attached with 9 downstreams, following is cli:

/usr/libexec/qemu-kvm \
-M q35 \
-monitor stdio \
-vnc :20 \
-m 4G \
-vga std \
-drive file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,unit=0,readonly=on \
-drive file=/home/OVMF_VARS.fd,if=pflash,format=raw,unit=1 \
-debugcon file:/home/q35.ovmf.log \
-global isa-debugcon.iobase=0x402 \
-device ioh3420,id=root.0,slot=1 \
-device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-device ioh3420,id=root.1,slot=2 \
-drive if=none,id=drive0,file=/home/pxb-ovmf.qcow2 \
-device virtio-blk-pci,drive=drive0,scsi=off,bus=root.1,disable-legacy=on,disable-modern=off \
-device ioh3420,id=root.2,slot=3 \
-device x3130-upstream,bus=root.2,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-device xio3130-downstream,bus=upstream1,id=downstream4,chassis=4 \
-device xio3130-downstream,bus=upstream1,id=downstream5,chassis=5 \
-device xio3130-downstream,bus=upstream1,id=downstream6,chassis=6 \
-device xio3130-downstream,bus=upstream1,id=downstream7,chassis=7 \
-device xio3130-downstream,bus=upstream1,id=downstream8,chassis=8 \
-device xio3130-downstream,bus=upstream1,id=downstream9,chassis=9 \

2. check the dmesg log when guest boot up


Actual results:
There have error message in the dmesg log:
[    0.673583] pci 0000:00:04.0: BAR 13: no space for [io  size 0x9000]
[    0.673584] pci 0000:00:04.0: BAR 13: failed to assign [io  size 0x9000]


Expected results:
Guest boot up successfully and no error message in the dmesg log

Additional info:
1. No error message when boot guest with 1 upstream which attached with downstreams
2. No error message when boot guest with 1 upstream which attached with 9 downstreams on seabios (seabios-1.9.1-4.el7.x86_64)
Comment 1 jingzhao 2016-06-22 06:27:48 UTC 
Created attachment 1170602 [details]
dmesg log
Comment 2 jingzhao 2016-06-22 06:29:08 UTC 
Created attachment 1170603 [details]
ovmf log
Comment 4 Marcel Apfelbaum 2016-06-26 12:26:52 UTC 
Hi,
Does it work with SeaBIOS?

Thanks,
Marcel
Comment 5 Marcel Apfelbaum 2016-06-26 12:27:55 UTC 
Please see comment #4.
Comment 6 jingzhao 2016-06-28 00:43:14 UTC 
(In reply to Marcel Apfelbaum from comment #5)
> Please see comment #4.

Hi Marcel

Please check the additional info of comment 0

No error message when boot guest with 1 upstream which attached with 9 downstreams on seabios (seabios-1.9.1-4.el7.x86_64)
Comment 7 Marcel Apfelbaum 2016-08-02 12:34:36 UTC 
Hi Laszlo,

Can you please have a look? I think you've already responded in another BZ, I just can't find it.

Thanks,
Marcel
Comment 8 Laszlo Ersek 2016-08-02 13:12:43 UTC 
My explanation here is going to be quite confusing, so please bear with me.

This BZ is reported against OVMF-20160608-1.git988715a.el7.noarch. That OVMF
version contains the fix (or, well, more precisely, mitigation) for bug
1333238. Said fix for bug 1333238 increases the IO space available to PCI IO
BAR allocations to 10*4KB, on Q35.

However, considering OVMF-20160608-1.git988715a.el7.noarch only, the fix for
bug 1333238 does *not* change the fact that OVMF does not allocate any
resources for bridges that are empty (= there are no devices behind them).
Looking at the OVMF log in comment 2, I can confirm this. To this part of
the QEMU command line:

> -device ioh3420,id=root.2,slot=3 \
> -device x3130-upstream,bus=root.2,id=upstream1 \
> -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
> -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
> -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
> -device xio3130-downstream,bus=upstream1,id=downstream4,chassis=4 \
> -device xio3130-downstream,bus=upstream1,id=downstream5,chassis=5 \
> -device xio3130-downstream,bus=upstream1,id=downstream6,chassis=6 \
> -device xio3130-downstream,bus=upstream1,id=downstream7,chassis=7 \
> -device xio3130-downstream,bus=upstream1,id=downstream8,chassis=8 \
> -device xio3130-downstream,bus=upstream1,id=downstream9,chassis=9 \

the following log snippet corresponds:

> PciBus: Resource Map for Bridge [00|04|00]
>
> PciBus: Resource Map for Bridge [03|00|00]
>
> PciBus: Resource Map for Bridge [04|00|00]
>
> PciBus: Resource Map for Bridge [04|01|00]
>
> PciBus: Resource Map for Bridge [04|02|00]
>
> PciBus: Resource Map for Bridge [04|03|00]
>
> PciBus: Resource Map for Bridge [04|04|00]
>
> PciBus: Resource Map for Bridge [04|05|00]
>
> PciBus: Resource Map for Bridge [04|06|00]
>
> PciBus: Resource Map for Bridge [04|07|00]
>
> PciBus: Resource Map for Bridge [04|08|00]

IOW, no resources allocated, because there are no devices there ultimately.

In such cases, Linux takes it upon itself to allocate resources for all
these bridges. And, of course, it runs out of available IO space. (We
discussed this elsewhere in a downstream mailing list thread.)

I don't know why Linux does this, and how it can be convinced not to do it.
I also don't know why the symptoms don't manifest with SeaBIOS.

My take is that SeaBIOS's hotplug preparation prevents it. Namely, SeaBIOS
allocates some IO space for all bridges in expectation of PCI(e) hotplug (is
that right?), so Linux is not tempted to do the resource allocation itself
for the bridges. Marcel, what's your opinion?

Note that OVMF's above-described behavior will change with the fix for bug
1332408. (That bug is in POST at the moment, and the fix is expected to show
up in the next official OVMF buiid.) Those patches change OVMF's behavior to
imitate SeaBIOS: OVMF is going to allocate resources for "empty" bridges as
well. Primarily for the sake of hotplug, but I believe it will also address
the symptoms seen here.

So, the fixes for bug 1333238 and bug 1332408 *in combination* should result
in the Linux symptoms to go away, up to and including 10 bridges in total
(including root ports, swith upstream ports, and switch downstream ports).
The exact command line in this bug report will likely not work (because it
requires more IO space than what's available in OVMF), but if the number of
bridges is decreased a little, then it should work fine with OVMF, assuming
it contains the fixes for both of said BZs.

I'll now implement a few metadata changes in this BZ; among them, I'll add
the TestOnly keyword. Let me emphasize again: the unmodified command line
from comment #0 will *not* work; the number of requested downstream ports
will have to be decreased. Each root port, switch upstream port, and switch
downstream port will likely take up 4KB of IO space, and OVMF only has room
for 10 * 4KB of IO ports.
Comment 9 Laszlo Ersek 2016-08-02 13:20:34 UTC 
Marcel, I also recall (vaguely!) some notes from you that it should be possible to tweak the bridges in QEMU, so that they don't demand IO space at all. I believe that with such device model changes, Linux might entirely abandon the idea of allocating IO space for the bridges, regardless of whether or not the firmware allocates IO space for the same.

If that's the avenue you'd like to pursue, then we can move this BZ back to qemu-kvm-rhev. Thanks!
Comment 11 Marcel Apfelbaum 2016-08-02 15:41:44 UTC 
Hi Laszlo,

(In reply to Laszlo Ersek from comment #8)
> My explanation here is going to be quite confusing, so please bear with me.
> 
> This BZ is reported against OVMF-20160608-1.git988715a.el7.noarch. That OVMF
> version contains the fix (or, well, more precisely, mitigation) for bug
> 1333238. Said fix for bug 1333238 increases the IO space available to PCI IO
> BAR allocations to 10*4KB, on Q35.
> 
> However, considering OVMF-20160608-1.git988715a.el7.noarch only, the fix for
> bug 1333238 does *not* change the fact that OVMF does not allocate any
> resources for bridges that are empty (= there are no devices behind them).
> Looking at the OVMF log in comment 2, I can confirm this.

OK, this is what I thought.

 To this part of
> the QEMU command line:
> 
> > -device ioh3420,id=root.2,slot=3 \
> > -device x3130-upstream,bus=root.2,id=upstream1 \
> > -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
> > -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
> > -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
> > -device xio3130-downstream,bus=upstream1,id=downstream4,chassis=4 \
> > -device xio3130-downstream,bus=upstream1,id=downstream5,chassis=5 \
> > -device xio3130-downstream,bus=upstream1,id=downstream6,chassis=6 \
> > -device xio3130-downstream,bus=upstream1,id=downstream7,chassis=7 \
> > -device xio3130-downstream,bus=upstream1,id=downstream8,chassis=8 \
> > -device xio3130-downstream,bus=upstream1,id=downstream9,chassis=9 \
> 
> the following log snippet corresponds:
> 
> > PciBus: Resource Map for Bridge [00|04|00]
> >
> > PciBus: Resource Map for Bridge [03|00|00]
> >
> > PciBus: Resource Map for Bridge [04|00|00]
> >
> > PciBus: Resource Map for Bridge [04|01|00]
> >
> > PciBus: Resource Map for Bridge [04|02|00]
> >
> > PciBus: Resource Map for Bridge [04|03|00]
> >
> > PciBus: Resource Map for Bridge [04|04|00]
> >
> > PciBus: Resource Map for Bridge [04|05|00]
> >
> > PciBus: Resource Map for Bridge [04|06|00]
> >
> > PciBus: Resource Map for Bridge [04|07|00]
> >
> > PciBus: Resource Map for Bridge [04|08|00]
> 
> IOW, no resources allocated, because there are no devices there ultimately.
> 
> In such cases, Linux takes it upon itself to allocate resources for all
> these bridges. And, of course, it runs out of available IO space. (We
> discussed this elsewhere in a downstream mailing list thread.)
> 
> I don't know why Linux does this, and how it can be convinced not to do it.
> I also don't know why the symptoms don't manifest with SeaBIOS.
> 

It must be because of the different allocation policy, but I don't know
the Linux policy yet.

> My take is that SeaBIOS's hotplug preparation prevents it. Namely, SeaBIOS
> allocates some IO space for all bridges in expectation of PCI(e) hotplug (is
> that right?), so Linux is not tempted to do the resource allocation itself
> for the bridges. Marcel, what's your opinion?

This is also my opinion. I implemented the SeaBIOS allocation policy to
allocate minimum SPEC allowed IO and MEM *if is not a pcie port.

> 
> Note that OVMF's above-described behavior will change with the fix for bug
> 1332408. (That bug is in POST at the moment, and the fix is expected to show
> up in the next official OVMF buiid.) Those patches change OVMF's behavior to
> imitate SeaBIOS: OVMF is going to allocate resources for "empty" bridges as
> well. Primarily for the sake of hotplug, but I believe it will also address
> the symptoms seen here.
> 

Good.

> So, the fixes for bug 1333238 and bug 1332408 *in combination* should result
> in the Linux symptoms to go away, up to and including 10 bridges in total
> (including root ports, swith upstream ports, and switch downstream ports).
> The exact command line in this bug report will likely not work (because it
> requires more IO space than what's available in OVMF), but if the number of
> bridges is decreased a little, then it should work fine with OVMF, assuming
> it contains the fixes for both of said BZs.
> 

Then we will add a maximum number of supported bridges for OVMF and change this
BZ command line and we will also add release notes.

> I'll now implement a few metadata changes in this BZ; among them, I'll add
> the TestOnly keyword. Let me emphasize again: the unmodified command line
> from comment #0 will *not* work; the number of requested downstream ports
> will have to be decreased. Each root port, switch upstream port, and switch
> downstream port will likely take up 4KB of IO space, and OVMF only has room
> for 10 * 4KB of IO ports.

Thanks,
Marcel
Comment 12 Marcel Apfelbaum 2016-08-02 15:45:30 UTC 
(In reply to Laszlo Ersek from comment #9)
> Marcel, I also recall (vaguely!) some notes from you that it should be
> possible to tweak the bridges in QEMU, so that they don't demand IO space at
> all. I believe that with such device model changes, Linux might entirely
> abandon the idea of allocating IO space for the bridges, regardless of
> whether or not the firmware allocates IO space for the same.
>

I don't have even a POC for the above and Paolo is not convinced of this direction. He thinks we should solve it in Linux kernel..

> If that's the avenue you'd like to pursue, then we can move this BZ back to
> qemu-kvm-rhev. Thanks!

Bottom line, I do want to pursue it, vut it is only an experiment, at this point
I prefer to wait until I'll know more.

Thanks Laszlo for the help!
Marcel
Comment 13 Laszlo Ersek 2017-01-27 14:11:05 UTC 
This BZ was actually fixed by fixing bug 1333238 and bug 1332408 (see comment 8). For some reason this one got stuck in "devel_ack?", which prevented it from progressing to ON_QA. Given that the necessary code changes have already been released, in RHEL-7.3, I'm closing this as CURRENTRELEASE.
Note You need to log in before you can comment on or make changes to this bug.