Bug 577182

Summary: vxge: not enough MMIO resources for SR-IOV error
Product: Red Hat Enterprise Linux 5 Reporter: Masroor <masroor.vettuparambil>
Component: kernelAssignee: Don Dutile (Red Hat) <ddutile>
Status: CLOSED ERRATA QA Contact: Network QE <network-qe>
Severity: high Docs Contact:
Priority: high    
Version: 5.5CC: arpit.patel, bzeranski, chrisw, cward, hjia, jon.mason, ltroan, peterm, ram.vepa, srirama.sharma, syeghiay, xen-maint
Target Milestone: betaKeywords: FutureFeature, HardwareEnablement, OtherQA
Target Release: 5.6   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-2.6.18-219.el5 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-01-13 21:21:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 523341    
Bug Blocks: 557597, 570372, 580413, 580949    

Description Masroor 2010-03-26 12:03:12 UTC
Description of problem:

When loading a driver for a PCIe device which supports SR-IOV, the BARs
associated with the SR-IOV capability may not be reliably requested, even if
there is space.
The bug is similar (identical?) to 523341
(https://bugzilla.redhat.com/show_bug.cgi?id=523341)

Version-Release number of selected component (if applicable):

2.6.18-190.el5xen

How reproducible:

Platform dependent, but when it happens it happens 100% of the time.

Steps to Reproduce:
1. Enable SR-IOV in system BIOS
2. Boot RHEL-5.5 2.6.18-190.el5xen
3. Load relevant driver (for SR-IOV capable device)

Actual results:

Driver modprobe fails and produces an error like: 
“PCI: Failed to allocate mem resource #12:1800000@b4800000 for 0000:02:00.0
vxge 0000:02:00.0: not enough MMIO resources for SR-IOV”


This means the physical device (aka Physical Function or PF) will not work,
and, of course, no virtual devices (aka Virtual Function or VF) associated w/
that PF can be allocated.

Expected results:

Driver is successfully loaded and VFs are successfully allocated making both
the PF and its VFs functional.


Additional info:

Comment 1 Chris Wright 2010-03-26 18:25:12 UTC
The info in comment #1 mention Xen, however this is not a Xen specific bug.  Neterion have shown same issue on bare metal case, and both bare metal and Xen are using the same PCI enumeration and resource reservation code.  I have updated the bug to be against kernel, and made it dependent on BZ# 523341 which is a generic entry for this resource allocation issue.

Masroor, can you please verify here in this BZ that the patched kernel available at:

http://et.redhat.com/~chrisw/rhel5/5.5/bz523341/

fixes the issue for you?

Comment 2 Masroor 2010-03-29 12:11:00 UTC
Hi Chris,

I have upgraded to kernel-xen-2.6.18-190.el5.bz523341.x86_64.rpm which fixes this issue.

Thanks,
Masroor

Comment 3 Ramkrishna Vepa 2010-04-14 02:49:37 UTC
Hi Chris,
Can this fix be included as part of a 5.5.z errata and 5.6? This will enable users to run the X3100 with SRIOV enabled.

Thanks,
Ram

Comment 4 Larry Troan 2010-05-19 16:34:58 UTC
Chris, is this patch already included in the kernel planned for 5.6? If not, are there plans to submit it and we will likely require an exception to have it considered.

This bug fix is a pre-req to a driver update required my some high-profile customers that plan to use the Exar NIC in their Red Hat certified servers.

Comment 5 Larry Troan 2010-06-15 15:38:22 UTC
Per Ram Vepa in a telecon 6/14, patches submitted and accepted upstream. They have been running a patched kernel from Chris Wright since last September.

Comment 6 Larry Troan 2010-07-27 21:12:34 UTC
Patches need to be attached to this bug by 8/01.

Comment 7 Larry Troan 2010-07-28 01:34:05 UTC
Chris, do we have the patches POSTed to rhkernel and approved for 5.6?

Note that the patch submission date is Aug 13 (not Aug 1 above) but they must be reviewed and approved by that date.

Comment 8 Larry Troan 2010-07-29 22:07:29 UTC
Requesting exception until NEEDINFO satisfied.

Comment 11 Larry Troan 2010-08-30 01:38:31 UTC
In response to comment #7,
Don Dutile has 3-part patch set he's working for 5.6. Removing NEEDINFO.

Comment 16 Srirama 2010-11-25 12:16:48 UTC
Installed RHEL 5.6 Beta (snapshot1) image and verified that the driver (version
2.0.8.20182-k) loads successfully in SRIOV mode. PF and all the VFs get
enumerated properly and are functional. Also verified that the passthrough
configuration works by attaching the PCI functions to 2 VMs and running traffic
between the VMs and also to the remote machine.

[root@linux-rhel-5 ~]# uname -a
Linux linux-rhel-5.6-181 2.6.18-231.el5xen #1 SMP Mon Nov 8 18:45:02 EST 2010
x86_64 x86_64 x86_64 GNU/Linux
[root@linux-rhel-5 ~]# lspci -d17d5:*
02:00.0 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:00.1 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:00.2 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:00.3 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:00.4 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:00.5 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:00.6 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:00.7 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:01.0 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:01.1 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:01.2 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:01.3 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:01.4 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:01.5 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:01.6 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:01.7 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)
02:02.0 Ethernet controller: Exar Corp. X3100 Series 10 Gigabit Ethernet PCIe
(rev 02)

Comment 19 errata-xmlrpc 2011-01-13 21:21:49 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0017.html