Bug 1348798

Created attachment 1170602 [details]
dmesg log

Created attachment 1170603 [details]
ovmf log

Hi,
Does it work with SeaBIOS?

Thanks,
Marcel

Please see comment #4.

(In reply to Marcel Apfelbaum from comment #5)
> Please see comment #4.

Hi Marcel

Please check the additional info of comment 0

No error message when boot guest with 1 upstream which attached with 9 downstreams on seabios (seabios-1.9.1-4.el7.x86_64)

Hi Laszlo,

Can you please have a look? I think you've already responded in another BZ, I just can't find it.

Thanks,
Marcel

My explanation here is going to be quite confusing, so please bear with me.

This BZ is reported against OVMF-20160608-1.git988715a.el7.noarch. That OVMF
version contains the fix (or, well, more precisely, mitigation) for bug
1333238. Said fix for bug 1333238 increases the IO space available to PCI IO
BAR allocations to 10*4KB, on Q35.

However, considering OVMF-20160608-1.git988715a.el7.noarch only, the fix for
bug 1333238 does *not* change the fact that OVMF does not allocate any
resources for bridges that are empty (= there are no devices behind them).
Looking at the OVMF log in comment 2, I can confirm this. To this part of
the QEMU command line:

> -device ioh3420,id=root.2,slot=3 \
> -device x3130-upstream,bus=root.2,id=upstream1 \
> -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
> -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
> -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
> -device xio3130-downstream,bus=upstream1,id=downstream4,chassis=4 \
> -device xio3130-downstream,bus=upstream1,id=downstream5,chassis=5 \
> -device xio3130-downstream,bus=upstream1,id=downstream6,chassis=6 \
> -device xio3130-downstream,bus=upstream1,id=downstream7,chassis=7 \
> -device xio3130-downstream,bus=upstream1,id=downstream8,chassis=8 \
> -device xio3130-downstream,bus=upstream1,id=downstream9,chassis=9 \

the following log snippet corresponds:

> PciBus: Resource Map for Bridge [00|04|00]
>
> PciBus: Resource Map for Bridge [03|00|00]
>
> PciBus: Resource Map for Bridge [04|00|00]
>
> PciBus: Resource Map for Bridge [04|01|00]
>
> PciBus: Resource Map for Bridge [04|02|00]
>
> PciBus: Resource Map for Bridge [04|03|00]
>
> PciBus: Resource Map for Bridge [04|04|00]
>
> PciBus: Resource Map for Bridge [04|05|00]
>
> PciBus: Resource Map for Bridge [04|06|00]
>
> PciBus: Resource Map for Bridge [04|07|00]
>
> PciBus: Resource Map for Bridge [04|08|00]

IOW, no resources allocated, because there are no devices there ultimately.

In such cases, Linux takes it upon itself to allocate resources for all
these bridges. And, of course, it runs out of available IO space. (We
discussed this elsewhere in a downstream mailing list thread.)

I don't know why Linux does this, and how it can be convinced not to do it.
I also don't know why the symptoms don't manifest with SeaBIOS.

My take is that SeaBIOS's hotplug preparation prevents it. Namely, SeaBIOS
allocates some IO space for all bridges in expectation of PCI(e) hotplug (is
that right?), so Linux is not tempted to do the resource allocation itself
for the bridges. Marcel, what's your opinion?

Note that OVMF's above-described behavior will change with the fix for bug
1332408. (That bug is in POST at the moment, and the fix is expected to show
up in the next official OVMF buiid.) Those patches change OVMF's behavior to
imitate SeaBIOS: OVMF is going to allocate resources for "empty" bridges as
well. Primarily for the sake of hotplug, but I believe it will also address
the symptoms seen here.

So, the fixes for bug 1333238 and bug 1332408 *in combination* should result
in the Linux symptoms to go away, up to and including 10 bridges in total
(including root ports, swith upstream ports, and switch downstream ports).
The exact command line in this bug report will likely not work (because it
requires more IO space than what's available in OVMF), but if the number of
bridges is decreased a little, then it should work fine with OVMF, assuming
it contains the fixes for both of said BZs.

I'll now implement a few metadata changes in this BZ; among them, I'll add
the TestOnly keyword. Let me emphasize again: the unmodified command line
from comment #0 will *not* work; the number of requested downstream ports
will have to be decreased. Each root port, switch upstream port, and switch
downstream port will likely take up 4KB of IO space, and OVMF only has room
for 10 * 4KB of IO ports.

Marcel, I also recall (vaguely!) some notes from you that it should be possible to tweak the bridges in QEMU, so that they don't demand IO space at all. I believe that with such device model changes, Linux might entirely abandon the idea of allocating IO space for the bridges, regardless of whether or not the firmware allocates IO space for the same.

If that's the avenue you'd like to pursue, then we can move this BZ back to qemu-kvm-rhev. Thanks!

Hi Laszlo,

(In reply to Laszlo Ersek from comment #8)
> My explanation here is going to be quite confusing, so please bear with me.
> 
> This BZ is reported against OVMF-20160608-1.git988715a.el7.noarch. That OVMF
> version contains the fix (or, well, more precisely, mitigation) for bug
> 1333238. Said fix for bug 1333238 increases the IO space available to PCI IO
> BAR allocations to 10*4KB, on Q35.
> 
> However, considering OVMF-20160608-1.git988715a.el7.noarch only, the fix for
> bug 1333238 does *not* change the fact that OVMF does not allocate any
> resources for bridges that are empty (= there are no devices behind them).
> Looking at the OVMF log in comment 2, I can confirm this.

OK, this is what I thought.

 To this part of
> the QEMU command line:
> 
> > -device ioh3420,id=root.2,slot=3 \
> > -device x3130-upstream,bus=root.2,id=upstream1 \
> > -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
> > -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
> > -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
> > -device xio3130-downstream,bus=upstream1,id=downstream4,chassis=4 \
> > -device xio3130-downstream,bus=upstream1,id=downstream5,chassis=5 \
> > -device xio3130-downstream,bus=upstream1,id=downstream6,chassis=6 \
> > -device xio3130-downstream,bus=upstream1,id=downstream7,chassis=7 \
> > -device xio3130-downstream,bus=upstream1,id=downstream8,chassis=8 \
> > -device xio3130-downstream,bus=upstream1,id=downstream9,chassis=9 \
> 
> the following log snippet corresponds:
> 
> > PciBus: Resource Map for Bridge [00|04|00]
> >
> > PciBus: Resource Map for Bridge [03|00|00]
> >
> > PciBus: Resource Map for Bridge [04|00|00]
> >
> > PciBus: Resource Map for Bridge [04|01|00]
> >
> > PciBus: Resource Map for Bridge [04|02|00]
> >
> > PciBus: Resource Map for Bridge [04|03|00]
> >
> > PciBus: Resource Map for Bridge [04|04|00]
> >
> > PciBus: Resource Map for Bridge [04|05|00]
> >
> > PciBus: Resource Map for Bridge [04|06|00]
> >
> > PciBus: Resource Map for Bridge [04|07|00]
> >
> > PciBus: Resource Map for Bridge [04|08|00]
> 
> IOW, no resources allocated, because there are no devices there ultimately.
> 
> In such cases, Linux takes it upon itself to allocate resources for all
> these bridges. And, of course, it runs out of available IO space. (We
> discussed this elsewhere in a downstream mailing list thread.)
> 
> I don't know why Linux does this, and how it can be convinced not to do it.
> I also don't know why the symptoms don't manifest with SeaBIOS.
> 

It must be because of the different allocation policy, but I don't know
the Linux policy yet.

> My take is that SeaBIOS's hotplug preparation prevents it. Namely, SeaBIOS
> allocates some IO space for all bridges in expectation of PCI(e) hotplug (is
> that right?), so Linux is not tempted to do the resource allocation itself
> for the bridges. Marcel, what's your opinion?

This is also my opinion. I implemented the SeaBIOS allocation policy to
allocate minimum SPEC allowed IO and MEM *if is not a pcie port.

> 
> Note that OVMF's above-described behavior will change with the fix for bug
> 1332408. (That bug is in POST at the moment, and the fix is expected to show
> up in the next official OVMF buiid.) Those patches change OVMF's behavior to
> imitate SeaBIOS: OVMF is going to allocate resources for "empty" bridges as
> well. Primarily for the sake of hotplug, but I believe it will also address
> the symptoms seen here.
> 

Good.

> So, the fixes for bug 1333238 and bug 1332408 *in combination* should result
> in the Linux symptoms to go away, up to and including 10 bridges in total
> (including root ports, swith upstream ports, and switch downstream ports).
> The exact command line in this bug report will likely not work (because it
> requires more IO space than what's available in OVMF), but if the number of
> bridges is decreased a little, then it should work fine with OVMF, assuming
> it contains the fixes for both of said BZs.
> 

Then we will add a maximum number of supported bridges for OVMF and change this
BZ command line and we will also add release notes.

> I'll now implement a few metadata changes in this BZ; among them, I'll add
> the TestOnly keyword. Let me emphasize again: the unmodified command line
> from comment #0 will *not* work; the number of requested downstream ports
> will have to be decreased. Each root port, switch upstream port, and switch
> downstream port will likely take up 4KB of IO space, and OVMF only has room
> for 10 * 4KB of IO ports.

Thanks,
Marcel

(In reply to Laszlo Ersek from comment #9)
> Marcel, I also recall (vaguely!) some notes from you that it should be
> possible to tweak the bridges in QEMU, so that they don't demand IO space at
> all. I believe that with such device model changes, Linux might entirely
> abandon the idea of allocating IO space for the bridges, regardless of
> whether or not the firmware allocates IO space for the same.
>

I don't have even a POC for the above and Paolo is not convinced of this direction. He thinks we should solve it in Linux kernel..

> If that's the avenue you'd like to pursue, then we can move this BZ back to
> qemu-kvm-rhev. Thanks!

Bottom line, I do want to pursue it, vut it is only an experiment, at this point
I prefer to wait until I'll know more.

Thanks Laszlo for the help!
Marcel

This BZ was actually fixed by fixing bug 1333238 and bug 1332408 (see comment 8). For some reason this one got stuck in "devel_ack?", which prevented it from progressing to ON_QA. Given that the necessary code changes have already been released, in RHEL-7.3, I'm closing this as CURRENTRELEASE.