Bug 493152

Summary: [Intel 5.4 FEAT] virtualization feature SR/IOV: kernel changes
Product: Red Hat Enterprise Linux 5 Reporter: Don Dugger <ddugger>
Component: kernelAssignee: Don Dugger <ddugger>
Status: CLOSED ERRATA QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: low    
Version: 5.4CC: dzickus, jane.lv, jjarvis, jvillalo, knoel, llim, luyu, syeghiay, xen-maint
Target Milestone: rcKeywords: FutureFeature
Target Release: 5.4   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 08:55:32 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 480376    
Attachments:
Description Flags
[1/17] BZ493152: Backport: PCI: rewrite PCI BAR reading code
none
[2/17] Backport: PCI: handle 64-bit resources better on 32-bit machines
none
[3/17] Backport: PCI: fix 64-vbit prefetchable memory resource BARs
none
[4/17] Backport: PCI: export __pci_read_base()
none
[5/17] Backport: PCI: support PCIe ARI capability
none
[6/17] Backport: PCI: fix ARI code to be compatible with mixed ARI/non-ARI systems
none
[7/17] Backport: PCI: enhance pci_ari_enabled()
none
[8/17] Backport: PCI: allow pci_alloc_child_bus() to handle a NULL bridge
none
[9/17] Backport: PCI: add a new function to map BAR offsets
none
[10/17] Backport: PCI: initialize and release SR-IOV capability
none
[11/17] Backport: PCI: restore saved SR-IOV state
none
[12/17] Backport: PCI: reserve bus range for SR-IOV device
none
[13/17] Backport: PCI: centralize device setup code
none
[14/17] Backport: PCI: add SR-IOV API for Physical Function driver
none
[15/17] Backport: PCI: Restore PCI Express capability registers after PM event
none
[16/17] Backport: PCI: save and restore PCIe 2.0 registers
none
[17/17] enable CONFIG_PCI_IOV none

Description Don Dugger 2009-03-31 19:44:29 UTC
Description of problem:

Kernel patches that implement SR/IOV in the Xen kernel

Comment 1 Don Dugger 2009-04-02 20:12:09 UTC
Created attachment 337900 [details]
[1/17] BZ493152: Backport: PCI: rewrite PCI BAR reading code

Upstream status: commit 6ac665c63dcac8fcec534a1d224ecbb8b867ad59
    Author: Matthew Wilcox <matthew>
    Date:   Mon Jul 28 13:38:59 2008 -0400

    PCI: rewrite PCI BAR reading code

    Factor out the code to read one BAR from the loop in pci_read_bases into
    a new function, __pci_read_base.  The new code is slightly more
    readable, better commented and removes the ifdef.

    Signed-off-by: Matthew Wilcox <willy.com>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Yu Zhao <yu.zhao>
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 2 Don Dugger 2009-04-02 20:14:03 UTC
Created attachment 337901 [details]
[2/17] Backport: PCI: handle 64-bit resources better on 32-bit machines

Upstream status: commit cc5499c3a607a392e8a7adb934aaf14b2c6a3519
    Author: Matthew Wilcox <matthew>
    Date:   Mon Jul 28 13:39:00 2008 -0400

    PCI: handle 64-bit resources better on 32-bit machines

    If the kernel is configured to support 64-bit resources on a 32-bit 
    machine, we can support 64-bit BARs properly.  Just change the condition
    to check sizeof(resource_size_t) instead of BITS_PER_LONG.

    Signed-off-by: Matthew Wilcox <willy.com>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Yu Zhao <yu.zhao>
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 3 Don Dugger 2009-04-02 20:15:38 UTC
Created attachment 337902 [details]
[3/17] Backport: PCI: fix 64-vbit prefetchable memory resource BARs

Upstream status: commit e354597cce8d219d135d65e585dc4f30323486b9
    Author: Peter Chubb <peterc.edu.au>
    Date:   Mon Oct 13 11:49:04 2008 +1100

    PCI: fix 64-vbit prefetchable memory resource BARs

    Since patch 6ac665c63dcac8fcec534a1d224ecbb8b867ad59 my infiniband
    controller hasn't worked.  This is because it has 64-bit prefetchable
    memory, which was mistakenly being  taken to be 32-bit memory.  The
    resource flags in this case are PCI_BASE_ADDRESS_MEM_TYPE_64 |
    PCI_BASE_ADDRESS_MEM_PREFETCH.
    
    This patch checks only for the PCI_BASE_ADDRESS_MEM_TYPE_64 bit; thus
    whether the region is prefetchable or not is ignored.  This fixes my
    Infiniband.

    Reviewed-by: Matthew Wilcox <matthew>
    Signed-off-by: Peter Chubb <peterc.edu.au>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Yu Zhao <yu.zhao>
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 4 Don Dugger 2009-04-02 20:16:55 UTC
Created attachment 337903 [details]
[4/17] Backport: PCI: export __pci_read_base()

Upstream status: commit 0b400c7ed4d027e02f6231afa39852a2d48e6f25
    Author: Yu Zhao <yu.zhao>
    Date:   Sat Nov 22 02:40:40 2008 +0800
    
    PCI: export __pci_read_base()

    Export __pci_read_base() so it can be used by whole PCI subsystem.
    
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 5 Don Dugger 2009-04-02 20:18:31 UTC
Created attachment 337904 [details]
[5/17] Backport: PCI: support PCIe ARI capability

Upstream status: commit 58c3a727cb73b75a9104d295f096cca12959a5a5
    Author: Yu Zhao <yu.zhao>
    Date:   Tue Oct 14 14:02:53 2008 +0800
    
    PCI: support PCIe ARI capability
    
    This patch adds support for PCI Express Alternative Routing-ID
    Interpretation (ARI) capability.
    
    The ARI capability extends the Function Number field of the PCI Express
    Endpoint by reusing the Device Number which is otherwise hardwired to 0.
    With ARI, an Endpoint can have up to 256 functions.

    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>
 
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 6 Don Dugger 2009-04-02 20:20:13 UTC
Created attachment 337905 [details]
[6/17] Backport: PCI: fix ARI code to be compatible with mixed ARI/non-ARI systems

Upstream status: commit 8113587c2d14d3be2414190845b2e2617c0aa33b
    Author: Zhao, Yu <yu.zhao>
    Date:   Thu Oct 23 13:15:39 2008 +0800
    
    PCI: fix ARI code to be compatible with mixed ARI/non-ARI systems

    The original ARI support code has a compatibility problem with non-ARI
    devices.  If a device doesn't support ARI, turning on ARI forwarding on
    its upper level bridge will cause undefined behavior.
 
    This fix turns on ARI forwarding only when the subordinate devices
    support it.
 
    Tested-by: Suresh Siddha <suresh.b.siddha>
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 7 Don Dugger 2009-04-02 20:21:11 UTC
Created attachment 337906 [details]
[7/17] Backport: PCI: enhance pci_ari_enabled()

Upstream status: commit 6a49d8120021897e139641062236215aac5d220e
    Author: Yu Zhao <yu.zhao>
    Date:   Sat Nov 22 02:38:21 2008 +0800
    
    PCI: enhance pci_ari_enabled()
    
    Change parameter of pci_ari_enabled() from 'pci_dev' to 'pci_bus'.

    ARI forwarding on the bridge mostly concerns the subordinate devices
    rather than the bridge itself. So this change will make the function
    easier to use.
 
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 9 Don Dugger 2009-04-02 22:17:15 UTC
Created attachment 337937 [details]
[8/17] Backport: PCI: allow pci_alloc_child_bus() to handle a NULL bridge

Upstream status: commit 3789fa8a2e534523c896a32a9f27f78d52ad7d82
    Author: Yu Zhao <yu.zhao>
    Date:   Sat Nov 22 02:41:07 2008 +0800
    
    PCI: allow pci_alloc_child_bus() to handle a NULL bridge
    
    Allow pci_alloc_child_bus() to allocate buses without bridge devices.
    Some SR-IOV devices can occupy more than one bus number, but there is no
    explicit bridges because that have internal routing mechanism.
    
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>
    
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 10 Don Dugger 2009-04-02 22:18:26 UTC
Created attachment 337938 [details]
[9/17] Backport: PCI: add a new function to map BAR offsets

Upstream status: commit 613e7ed6f72b1a115f7ece8ce1b66cf095de1348
    Author: Yu Zhao <yu.zhao>
    Date:   Sat Nov 22 02:41:27 2008 +0800
    
    PCI: add a new function to map BAR offsets 
    
    Add a function to map a given resource number to a corresponding
    register so drivers can get the offset and type of device specific BARs.
    
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>
    
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 11 Don Dugger 2009-04-02 22:19:43 UTC
Created attachment 337939 [details]
[10/17] Backport: PCI: initialize and release SR-IOV capability

Upstream status: commit d1b054da8f599905f3c18a218961dcf17f9d5f13
    Author: Yu Zhao <yu.zhao>
    Date:   Fri Mar 20 11:25:11 2009 +0800
    
    PCI: initialize and release SR-IOV capability
    
    If a device has the SR-IOV capability, initialize it (set the ARI
    Capable Hierarchy in the lowest numbered PF if necessary; calculate
    the System Page Size for the VF MMIO, probe the VF Offset, Stride
    and BARs). A lock for the VF bus allocation is also initialized if
    a PF is the lowest numbered PF.
    
    Reviewed-by: Matthew Wilcox <willy.com>
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 12 Don Dugger 2009-04-02 22:20:41 UTC
Created attachment 337940 [details]
[11/17] Backport: PCI: restore saved SR-IOV state

Upstream status: commit 8c5cdb6adc6688b9b8fd82ea4a5cf4674dabad79
    Author: Yu Zhao <yu.zhao>
    Date:   Fri Mar 20 11:25:12 2009 +0800
    
    PCI: restore saved SR-IOV state

    Restore the volatile registers in the SR-IOV capability after the
    D3->D0 transition.

    Reviewed-by: Matthew Wilcox <willy.com>
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>
 
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 13 Don Dugger 2009-04-02 22:22:00 UTC
Created attachment 337942 [details]
[12/17] Backport: PCI: reserve bus range for SR-IOV device

Upstream status: commit a28724b0fb909d247229a70761c90bb37b13366a
    Author: Yu Zhao <yu.zhao>
    Date:   Fri Mar 20 11:25:13 2009 +0800
    
    PCI: reserve bus range for SR-IOV device

    Reserve the bus number range used by the Virtual Function when
    pcibios_assign_all_busses() returns true.

    Reviewed-by: Matthew Wilcox <willy.com>
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>
 
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 14 Don Dugger 2009-04-02 22:23:03 UTC
Created attachment 337943 [details]
[13/17] Backport: PCI: centralize device setup code

Upstream status: commit 480b93b7837fb3cf0579a42f4953ac463a5b9e1e
    Author: Yu Zhao <yu.zhao>
    Date:   Fri Mar 20 11:25:14 2009 +0800
    
    PCI: centralize device setup code 

    Move the device setup stuff into pci_setup_device() which will be used
    to setup the Virtual Function later.

    Reviewed-by: Matthew Wilcox <willy.com>
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>
 
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 15 Don Dugger 2009-04-02 22:24:01 UTC
Created attachment 337944 [details]
[14/17] Backport: PCI: add SR-IOV API for Physical Function driver

Upstream status: commit dd7cc44d0bcec5e9c42fe52e88dc254ae62eac8d
    Author: Yu Zhao <yu.zhao>
    Date:   Fri Mar 20 11:25:15 2009 +0800
    
    PCI: add SR-IOV API for Physical Function driver

    Add or remove the Virtual Function when the SR-IOV is enabled or
    disabled by the device driver. This can happen anytime rather than
    only at the device probe stage.
    
    Reviewed-by: Matthew Wilcox <willy.com>
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Gerd Hoffman <kraxel>   
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 16 Don Dugger 2009-04-02 22:25:13 UTC
Created attachment 337945 [details]
[15/17] Backport: PCI: Restore PCI Express capability registers after PM event

Upstream status: commit b56a5a23bfecd9cac9187164a9d5f22d287c48b9
    Author: Michael S. Tsirkin <mst.il>
    Date:   Mon Aug 21 16:22:22 2006 +0300
    
    PCI: Restore PCI Express capability registers after PM event

    Restore PCI Express capability registers after PM event.
    This includes maxumum MTU for PCI express and other vital data.
    
    Signed-off-by: Michael S. Tsirkin <mst.il>
    Signed-off-by: Greg Kroah-Hartman <gregkh>
    
Signed-off-by: Yu Zhao <yu.zhao>
Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 17 Don Dugger 2009-04-02 22:26:15 UTC
Created attachment 337946 [details]
[16/17] Backport: PCI: save and restore PCIe 2.0 registers

Upstream status: commit 898585172fa729513d8636257b44bd1cfd279096
    Author: Yu Zhao <yu.zhao>
    Date:   Mon Feb 16 02:55:47 2009 +0800
    
    PCI: save and restore PCIe 2.0 registers

    PCIe 2.0 defines several new registers (Device Control 2, Link Control 2,
    and Slot Control 2). Save and retore them in pci_save_pcie_state() and
    pci_restore_pcie_state().
    
    Signed-off-by: Yu Zhao <yu.zhao>
    Signed-off-by: Jesse Barnes <jbarnes>

Signed-off-by: Gerd Hoffman <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 18 Don Dugger 2009-04-02 22:27:16 UTC
Created attachment 337947 [details]
[17/17] enable CONFIG_PCI_IOV

Add the new config option created by the SR/IOV patches.

Signed-off-by: Gerd Hoffmann <kraxel>
Signed-off-by: Don Dugger <donald.d.dugger>

Comment 19 Don Zickus 2009-05-14 19:35:19 UTC
in kernel-2.6.18-148.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please do NOT transition this bugzilla state to VERIFIED until our QE team
has sent specific instructions indicating when to do so.  However feel free
to provide a comment indicating that this fix has been verified.

Comment 22 errata-xmlrpc 2009-09-02 08:55:32 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1243.html