Bug 1271457

# rpm -qa |grep seabios
seabios-bin-1.7.5-11.el7.noarch

This is not a bug, I'll explain why. The PCI/PCI bridge spec requires to assign to each PCI bridge 4K IO space, even if the devices behind the bridge require much less. Since the IO space is very limited (64K if I remember correctly) there are so much pci-bridges you may have.

However, if you need more than 9-10 pci-bridges you can nest them (bridge behind bridge). The reason it will work is that the nested bridges will share the same IO ranges as the bus 0 originator bridge.

Something like:


 `for i in {1..8}; do echo -device pci-bridge,chassis_nr=$i,id=b$i -device pci-bridge,chassis_nr=$((2*$i)),bus=b$i,addr=1; done`

This will allow 16 bridges and so on.

Marcel, it may not be possible for SeaBIOS to give the devices all the ioport space they need in this configuration, but don't you think we should fail more gracefully here?  4K may be the granularity at which we can program ioport space to a subordinate bus, but assigning no ioport space is also an option.  Clearly we won't be able to program ioport resources of those devices behind bridges that don't have any allocated to the bridge, but not all devices require ioport space.  Even in the physical world, a device my expose ioport space, but only require it in very limited use cases, such as for boot, but not for runtime support.

IMO there is a bug here and SeaBIOS should fail more gracefully, perhaps with a warning output to the console about exhausted ioport resources.  I would expect robust physical hardware to do the same rather than simply failing to boot.

(In reply to Alex Williamson from comment #4)
> Marcel, it may not be possible for SeaBIOS to give the devices all the
> ioport space they need in this configuration, but don't you think we should
> fail more gracefully here?  4K may be the granularity at which we can
> program ioport space to a subordinate bus, but assigning no ioport space is
> also an option.  Clearly we won't be able to program ioport resources of
> those devices behind bridges that don't have any allocated to the bridge,
> but not all devices require ioport space.  Even in the physical world, a
> device my expose ioport space, but only require it in very limited use
> cases, such as for boot, but not for runtime support.
> 

Hi Alex,
Thank you for having a look at this problem.

> IMO there is a bug here and SeaBIOS should fail more gracefully, perhaps
> with a warning output to the console about exhausted ioport resources.  I
> would expect robust physical hardware to do the same rather than simply
> failing to boot.

Once the IO space is finished the SeaBIOS fails with a panic message
that can be seen at least with the debug device enabled. By the way, this
mode of operation is the "standard" SeaBIOS way to halt when all
kind of resources can't be assigned, not only IO.

Michael had another idea regarding what SeaBIOS should do when no IO space
remains: instead of panic, just skip assigning IO space to those bridges.

I didn't quite like this approach because it is not a 'fair' way to assign
resources, maybe the devices behind the bridge that got the IO don't need it,
while others do need it even if they got skipped.

I preferred the approach: skip assigning IO to the bridges that devices
attached to them do not require IO.

Thanks,
Marcel

Hi Marcel,

(In reply to Marcel Apfelbaum from comment #5)
> (In reply to Alex Williamson from comment #4)
> > IMO there is a bug here and SeaBIOS should fail more gracefully, perhaps
> > with a warning output to the console about exhausted ioport resources.  I
> > would expect robust physical hardware to do the same rather than simply
> > failing to boot.
> 
> Once the IO space is finished the SeaBIOS fails with a panic message
> that can be seen at least with the debug device enabled. By the way, this
> mode of operation is the "standard" SeaBIOS way to halt when all
> kind of resources can't be assigned, not only IO.
> 
> Michael had another idea regarding what SeaBIOS should do when no IO space
> remains: instead of panic, just skip assigning IO space to those bridges.

Right, once ioport space is exhausted, simply stop assigning it, report error, and hobble along with a partially programmed PCI space.  The guest may be able to reallocate the buses with more knowledge of the specific devices or the guest drivers may not actually make use of the unprogrammed ioport space.

> I didn't quite like this approach because it is not a 'fair' way to assign
> resources, maybe the devices behind the bridge that got the IO don't need it,
> while others do need it even if they got skipped.

At the BIOS level we can only determine which devices are requesting ioport resources, whether they actually need it or not is for the option ROM and guest level drivers to decide.
 
> I preferred the approach: skip assigning IO to the bridges that devices
> attached to them do not require IO.

Sure, that seems like a bug in itself if SeaBIOS is wasting ioport resources where none is actually used by subordinate devices, but that's really a corner case to the problem of ioport space being exhausted.  This seems like a case where we could improve SeaBIOS to an "enterprise class" by adding robustness to fail gracefully and predictably in this case of resource exhaustion to achieve a better user experience.  Of course libvirt could also make a default configuration that gives SeaBIOS a better chance of success, per your suggestions, which is being addressed in bug 1320908 and led me to this one.  Thanks

(In reply to Alex Williamson from comment #6)
> Hi Marcel,
> 
> (In reply to Marcel Apfelbaum from comment #5)
> > (In reply to Alex Williamson from comment #4)
> > > IMO there is a bug here and SeaBIOS should fail more gracefully, perhaps
> > > with a warning output to the console about exhausted ioport resources.  I
> > > would expect robust physical hardware to do the same rather than simply
> > > failing to boot.
> > 
> > Once the IO space is finished the SeaBIOS fails with a panic message
> > that can be seen at least with the debug device enabled. By the way, this
> > mode of operation is the "standard" SeaBIOS way to halt when all
> > kind of resources can't be assigned, not only IO.
> > 
> > Michael had another idea regarding what SeaBIOS should do when no IO space
> > remains: instead of panic, just skip assigning IO space to those bridges.

Hi Alex,

> 
> Right, once ioport space is exhausted, simply stop assigning it, report
> error, and hobble along with a partially programmed PCI space.  The guest
> may be able to reallocate the buses with more knowledge of the specific
> devices or the guest drivers may not actually make use of the unprogrammed
> ioport space.
> 

OK, since both you and Michael think is a plausible solution, I can give it a try.
How do you propose to report the error? I didn't check it yet, but I think the
resources allocation step goes before initiating the vga drivers and the other
way is to report back to QEMU. However, the fwcfg stuff goes only one way.

> > I didn't quite like this approach because it is not a 'fair' way to assign
> > resources, maybe the devices behind the bridge that got the IO don't need it,
> > while others do need it even if they got skipped.
> 
> At the BIOS level we can only determine which devices are requesting ioport
> resources, whether they actually need it or not is for the option ROM and
> guest level drivers to decide.
>  
> > I preferred the approach: skip assigning IO to the bridges that devices
> > attached to them do not require IO.
> 
> Sure, that seems like a bug in itself if SeaBIOS is wasting ioport resources
> where none is actually used by subordinate devices, but that's really a
> corner case to the problem of ioport space being exhausted.  This seems like
> a case where we could improve SeaBIOS to an "enterprise class" by adding
> robustness to fail gracefully and predictably in this case of resource
> exhaustion to achieve a better user experience.  Of course libvirt could
> also make a default configuration that gives SeaBIOS a better chance of
> success, per your suggestions, which is being addressed in bug 1320908 and
> led me to this one.  Thanks

I get your point, we need a way to report back to QEMU, I'll speak with Gerd about it.
Thanks,
Marcel

What would we expect to happen on bare metal where we don't have a hypervisor?  The BIOS would probably spit out some warning on the console (default VGA) with a timeout or key press to continue.  I would think SeaBIOS could do something similar.  The error may occur prior to VGA init, but we don't necessarily need to post the warning immediately at the point we hit the exhaustion.  We could simply flag that it occurred, write out the details to the debug port, and splash the warning to the console after it posts.  I'm not sure what QEMU would do if it were notified of the error other than just an error_report().  Thanks