Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 506841

Summary: RHEL5.4 -154 e1000e using MSI-X hangs system
Product: Red Hat Enterprise Linux 5 Reporter: Don Dutile (Red Hat) <ddutile>
Component: kernelAssignee: Andy Gospodarek <agospoda>
Status: CLOSED ERRATA QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: urgent Docs Contact:
Priority: high    
Version: 5.4CC: ddutile, dzickus, emcnabb, john.ronciak, mgahagan, peterm, rpacheco, syeghiay
Target Milestone: rc   
Target Release: 5.4   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 08:15:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
e1000e-msix-fix.patch none

Description Don Dutile (Red Hat) 2009-06-18 20:59:59 UTC
Description of problem:
Booting RHEL5.4 -154 kernel with an e1000e device using MSI-X interrupts
hangs/wedges the system

Version-Release number of selected component (if applicable):
2.6.18-154.el5.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Boot Intel Weybridge machine with 82547L plug-in PCIe NIC card 
2.
3.
  
Actual results:
System wedges when it reaches:
"Determining IP information for eth1..."

Expected results:
System doesn't wedge at about boot/config point

Additional info:
Same configuration works on 5.2, 5.3 & earlier 5.4 kernels (up to -153).

Andy Gospodarek has diagnosed the problem on my Weybridge
machine and has proposed patch to fix.

Comment 1 Andy Gospodarek 2009-06-18 22:08:32 UTC
Created attachment 348575 [details]
e1000e-msix-fix.patch

Upstream patch (that should apply with fuzz) posted and already ACKed by Intel.

Comment 2 RHEL Program Management 2009-06-19 00:41:33 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 4 John Ronciak 2009-06-19 16:43:41 UTC
Testing of -154 is starting today for e1000e.  The ixgbe testing was started yesterday on this kernel and will be done later today.

We'll report back on the testing later today.

Comment 5 John Ronciak 2009-06-19 23:00:30 UTC
OK now I'm a bit confused here.  The 82547 is a PCI part and uses the e1000 driver.  So are you saying that you plug a PCIe NIC into one of these systems somehow?  Or is that the e1000 driver is being used and has an issue?

We found no problems in our testing of the -154 kernel e1000e driver.  It passed both BAT and stress testing.

Comment 6 Andy Gospodarek 2009-06-22 14:07:16 UTC
John, that is a typo -- this problem was seen on 82574.

Comment 7 Don Zickus 2009-06-22 15:35:57 UTC
in kernel-2.6.18-155.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please do NOT transition this bugzilla state to VERIFIED until our QE team
has sent specific instructions indicating when to do so.  However feel free
to provide a comment indicating that this fix has been verified.

Comment 9 John Ronciak 2009-06-22 16:15:39 UTC
OK,We'll start testing the new -155 kernel today.  This is very odd however as we run a whole bunch of tests on a number of different systems and didn't see this problem.  We'll try to find a Weybridge system (with an 82574 NIC) and test on it as well.

Comment 10 Andy Gospodarek 2009-06-22 16:33:34 UTC
(In reply to comment #9)
> OK,We'll start testing the new -155 kernel today.  This is very odd however as
> we run a whole bunch of tests on a number of different systems and didn't see
> this problem.  We'll try to find a Weybridge system (with an 82574 NIC) and
> test on it as well.  

If you use -154 on hardware/driver using MSI-X and you don't notice any hangs when bringing it up, check mpstat output and you will probably see that at least of the the cpus is doing nothing but handing softirqs since they are constantly running.

Comment 11 John Ronciak 2009-06-22 17:26:53 UTC
We reproduced the bug on the -154 kernel.  We tested only using MSI capable devices.  When we tried Hartwell (MSI-X) we see the problem.  The -155 kernel is now under test.

Comment 12 John Ronciak 2009-06-23 23:34:45 UTC
We have tested the -155 kernel and do not see the problem.  We also tested MSI and legacy interrupt mode which also work.  

We are testing a patch to this driver now in-house which may need to be applied to the 5.4 version of the e1000e driver.  We will know tomorrow.  If we do need it I'll make a full disclosure as to why it's needed and also make sure we push the patch upstream.

Comment 15 errata-xmlrpc 2009-09-02 08:15:10 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1243.html