Bug 1410578

Summary: PCIe: Add an option to PCIe ports to disable IO port space support
Product: Red Hat OpenStack Reporter: Marcel Apfelbaum <marcel>
Component: openstack-novaAssignee: OSP DFG:Compute <osp-dfg-compute>
Status: CLOSED INSUFFICIENT_DATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: low Docs Contact:
Priority: low    
Version: unspecifiedCC: ailan, berrange, chayang, dasmith, dyuan, eglynn, hhan, jinzhao, juzhang, kchamart, laine, lersek, libvirt-maint, lmen, lyarwood, marcel, michen, mtessun, sbauza, sferdjao, sgordon, srevivo, stephenfin, virt-maint, vromanso, xfu, xuzhang
Target Milestone: ---Keywords: FutureFeature, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 1408810 Environment:
Last Closed: 2018-08-28 09:49:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1344299, 1408810    
Bug Blocks: 1410577    

Description Marcel Apfelbaum 2017-01-05 19:27:00 UTC
+++ This bug was initially created as a clone of Bug #1408810 +++

+++ This bug was initially created as a clone of Bug #1344299 +++

Even if the firmware skips assigning IO ranges to PCIe ports (root
ports/downstream ports), Linux guests will still try to assign them IO.

We can to add a parameter "disable-io" to PCIe ports to disable IO support.
It will work by making IO base/limit registers read-only so both firmware
and guest OSes will comply.

--- Additional comment from RHEL Product and Program Management on 2016-06-09 08:04:38 EDT ---

Since this bug report was entered in bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Marcel Apfelbaum on 2016-06-23 06:07:07 EDT ---

We intend to provide a 'thin' version of Q35 for 7.3 to be used mainly with
virtio devices which are PCIe, the IO limitation will not be an issue.


Add libvirt support for the device command line parameters.

Comment 1 Marcel Apfelbaum 2017-01-05 19:30:27 UTC
By default libvirt will disable IO space support for PCI Express Root Ports since all PCI Express devices should work just fine using only MMIO.

However there are cases when we do want to enable IO space support:
1. We might want to assign a phys device (using vfio) and the device is not spec
   compliant and requires IO ports to function properly.
2. We might have a guest firmware with an ancient driver knowing only IO ports.

Decisions like this cannot be done by libvirt, we need the upper layer to decide which PCI Express Root Ports need to have IO ports enabled.

Comment 2 Stephen Gordon 2017-01-26 21:20:55 UTC
Marcel can you provide some detail on what the actual use case is driving this?

Comment 3 Marcel Apfelbaum 2017-02-13 13:46:09 UTC
(In reply to Stephen Gordon from comment #2)
> Marcel can you provide some detail on what the actual use case is driving
> this?

Hi Stephen,
I am sorry for the delayed response, I was in PTO.

The problem is the IO ports space is rather limited and for PCI Express machines each PCI Express Root Port allows only to a single (multi-function) device. Since each such port is actually a PCI bridge, the guest firmware and after that the guest OS will try to allocate some IO range for each.

That means that after ~10 PCI root ports are used all IO ports space is exhausted and then:
 1. The firmware may panic and halt (SeaBIOS does that).
 2. The OS may try to assign the IO space by itself and in the best scenario we only see some warnings on logs.

Since all PCI Express devices should be able to work without using IO ports we can instruct the PCI Express Root Ports to disable IO ports by default.

The interesting part is what happens if we need to plug an assigned device (vfio) to a Root Port and it requires IO ports to function properly?

Since now the Root Ports have IO space disabled by default, we need to 'enable' it by setting the corresponding property.

Thanks,
Marcel

Comment 4 Stephen Finucane 2018-08-28 09:49:19 UTC
There's no clear explanation as to what use cases this feature will resolve. Until such a time as this is provided, I'm going to mark this as closed.