Bug 754671

Summary: ixgbe: not enough MMIO resources for SR-IOV on 82599EB NIC
Product: Red Hat Enterprise Linux 5 Reporter: Zhouping Liu <zliu>
Component: kernelAssignee: Tony Camuso <tcamuso>
Status: CLOSED NOTABUG QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.8CC: bsarathy, ccui, ddutile, juzhang, knoel, leiwang, michen, nagananda.chumbalkar, nobody+295318, qguan, qwan, shuah.khan, tcamuso
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-10-10 18:15:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Zhouping Liu 2011-11-17 09:57:45 UTC
Description of problem:
when I executed "modprobe -r ixgbe; modprobe ixgbe max_vfs=2" to create SR-IOV VF,
these error messages occurred:
...
PCI: Enabling device 0000:11:00.0 (0040 -> 0042)
ixgbe 0000:11:00.0: not enough MMIO resources for SR-IOV
ixgbe 0000:11:00.0: (unregistered net_device): Failed to enable PCI sriov: -12
PCI: Enabling device 0000:11:00.1 (0040 -> 0042)
ixgbe 0000:11:00.1: not enough MMIO resources for SR-IOV
ixgbe 0000:11:00.1: (unregistered net_device): Failed to enable PCI sriov: -12

and I did the same actions on the two machines:
hp-dl580g7-01.rhts.eng.nay.redhat.com
sun-cp3270-01.rhts.eng.bos.redhat.com
the result was similar.
also I found the PCI messages when the system boot:
...
...
15200n81 rhgb quiet
   [Linux-bzImage, setup=0x1e00, size=0x20203c]
initrd /initrd-2.6.18-294.el5.img
   [Linux-initrd @ 0x37c49000, 0x3a6063 bytes]

Memory for crash kernel (0x0 to 0x0) notwithin permissible range
PCI: Failed to allocate mem resource #12:100000@0 for 0000:11:00.0
PCI: Failed to allocate mem resource #15:100000@0 for 0000:11:00.0
PCI: Failed to allocate mem resource #12:100000@0 for 0000:11:00.1
PCI: Failed to allocate mem resource #15:100000@0 for 0000:11:00.1
�Red Hat nash version 5.1.19.6 starting
  Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup00" using metadata type lvm2
  2 logical volume(s) in volume group "VolGroup00" now active
		Welcome to Red Hat Enterprise Linux Server
		Press 'I' to enter interactive startup.
Setting clock  (localtime): Thu Nov 17 03:56:13 EST 2011 [  OK  ]
Starting udev: [  OK  ]
Loading default keymap (us): [  OK  ]
Setting hostname hp-dl580g7-01.rhts.eng.nay.redhat.com:  [  OK  ]
Setting up Logical Volume Management:   2 logical volume(s) in volume group "VolGroup00" now active
...
...
FYI: I guessed the "PCI: Failed" maybe caused the bug.

Version-Release number of selected component (if applicable):
CPU: intel Xeon
os tree: RHEL5.8-Server-20111030.0
[root@hp-dl580g7-01 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 console=ttyS0,115200n81 intel_iommu=on rhgb quiet
[root@hp-dl580g7-01 ~]# lspci | grep Eth
04:00.0 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
04:00.1 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
04:00.2 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
04:00.3 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
0b:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (rev 02)
0b:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (rev 02)
11:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
11:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
[root@hp-dl580g7-01 ~]# uname -ri
2.6.18-294.el5 x86_64

How reproducible:


Steps to Reproduce:
1. add intel_iommu=on to kernel command line, reboot
2. # modprobe -r ixgbe; modprobe ixgbe max_vfs=number
3. # dmesg
  
Actual results:
...
ixgbe 0000:11:00.1: not enough MMIO resources for SR-IOV
ixgbe 0000:11:00.1: (unregistered net_device): Failed to enable PCI sriov: -12
ixgbe 0000:11:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 1
ixgbe 0000:11:00.1: (PCI Express:5.0GT/s:Width x8) 00:1b:21:4a:fe:99
ixgbe 0000:11:00.1: MAC: 2, PHY: 15, SFP+: 6, PBA No: E66562-003
ixgbe 0000:11:00.1: Intel(R) 10 Gigabit Network Connection
...
Expected results:
no the above messages

Additional info:

Comment 1 Don Dutile (Red Hat) 2011-11-17 14:21:53 UTC
This is due to a limitation in the system bios.
re-assigning to HP partner that supports these systems.

Comment 2 Tony Camuso 2011-11-17 16:09:16 UTC
HP is working on SR-IOV support for this platform. 

I may be able to obtain a side-built BIOS for testing.

Stay tuned  ...

Comment 3 Zhouping Liu 2011-11-18 01:50:05 UTC
I have tested on RHEL5.7GA, the issue existed.

Comment 4 Tony Camuso 2011-11-29 12:48:33 UTC
We are working on SR-IOV support for RHEL6 on this platform. 

Is RHEL5.8 scheduled to support SR-IOV, also?

Comment 5 Tony Camuso 2011-12-15 13:50:16 UTC
I have access to side-built BIOS for partner testing on select ProLiant G7 platforms and 10 Gb NICs.

Please contact me by email for specifics if you want a copy.

Comment 6 Tony Camuso 2012-01-10 14:23:49 UTC
The BIOS that supports SR-IOV for these platforms is not ready yet. I was mistaken in thinking that I could get a side-built BIOS this early. 

I will update this BZ as soon as I get such a BIOS.

Comment 7 Tony Camuso 2012-10-01 20:05:58 UTC
Is the BIOS going to be updated for DL580G7 to support SR-IOV?

Comment 8 Shuah Khan 2012-10-10 14:33:26 UTC
In our internal testing, it has been determined that this problem goes away when more memory is installed in the system. This bugzilla can be closed.

Comment 9 Tony Camuso 2012-10-10 18:15:26 UTC
Thanks, Shuah.

Closing as NOTABUG.