Bug 713351

Summary: [Sandy Bridge] e1000 IRQ being disabled resulting in 2MB/s network speeds.
Product: [Fedora] Fedora Reporter: Simon Lea <simon_lea>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 15CC: gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-07-11 17:49:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Simon Lea 2011-06-15 03:36:32 UTC 
Description of problem: I have just upgraded from an Intel C2D 775 processor and MB to a i3(2100) 1155 processor and MB and have been seeing terrible network speeds and sometimes terrible disk to disk transfer speeds.  This is affecting both Fedora 14 and Fedora 15.  I use an Intel GT 1000/Pro network card rather than the motherboard NIC.

On further investigation of the system log (/var/log/messages) I found the following;

irq 16: nobody cared (try booting with the "irqpoll" option)
Pid: 0, comm: swapper Not tainted 2.6.35.13-92.fc14.x86_64 #1
Call Trace:
<IRQ> [<ffffffff810a74f3>] __report_bad_irq.clone.1+0x3d/0x8b
[<ffffffff810a765b>] note_interrupt+0x11a/0x17f
[<ffffffff810a813b>] handle_fasteoi_irq+0xa8/0xce
[<ffffffff8100c2ea>] handle_irq+0x88/0x90
[<ffffffff81470b44>] do_IRQ+0x5c/0xb4
[<ffffffff8146b093>] ret_from_intr+0x0/0x11
<EOI> [<ffffffff8102b7dd>] ? native_safe_halt+0xb/0xd
[<ffffffff81290b75>] acpi_safe_halt+0x2a/0x43
[<ffffffff81290bae>] acpi_idle_do_entry+0x20/0x30
[<ffffffff81290c27>] acpi_idle_enter_c1+0x69/0xb6
[<ffffffff8146e01e>] ? notifier_call_chain+0x14/0x63
[<ffffffff81395d56>] ? menu_select+0x177/0x28c
[<ffffffff81394d6d>] cpuidle_idle_call+0x8b/0xe9
[<ffffffff81008325>] cpu_idle+0xaa/0xcc
[<ffffffff81452876>] rest_init+0x8a/0x8c
[<ffffffff81ba1c49>] start_kernel+0x40b/0x416
[<ffffffff81ba12c6>] x86_64_start_reservations+0xb1/0xb5
[<ffffffff81ba13c2>] x86_64_start_kernel+0xf8/0x107
handlers:
[<ffffffff813148a8>] (ata_bmdma_interrupt+0x0/0x1a)
[<ffffffffa00d8622>] (mpt_interrupt+0x0/0x890 [mptbase]) <--Sata PCIe controller
[<ffffffffa00d8622>] (mpt_interrupt+0x0/0x890 [mptbase]) <--Sata PCIe controller
[<ffffffffa0132850>] (e1000_intr+0x0/0xe9 [e1000]) <--- Network Card
Disabling IRQ #16 

After turning off the audio, USB3 and network controller on the MB I still got the same error (only the e1000 affected this time and a different IRQ number).  The error can occur at any time, not just on boot, but it usually happens fairly quickly.

Version-Release number of selected component (if applicable):
Fedora 14 - (2.6.35.13-92.fc14.x86_64) 
Latest Fedora 15 install CD (as of last week).

How reproducible:
Always

Steps to Reproduce:
1. Turn server on in above config.
2. Perform a large file transfer (5-10GB)
3. Watch for network transfer speed to change.
  
Actual results:
Network transfer speeds not able to go higher than 3MB/s
Disk to disk transfer speeds also sometimes affected depending on which devices were also using the same IRQ which was disabled.

Expected results:
Normal boot, network speeds of approx 80-90MB/s (obtained when using the previous Intel C2D MB and CPU).

Additional info:

Hardware list;
Intel i3 2100 CPU (LGA1155)
Asus P8H67-V (v3) MB
4Gb DDR3 Ram 
2xLSI 1068e PCIe (8 sata drive controllers)
1xIntel GT 1000/Pro (PCI) network controller.
Various hard drives
Norco 4220 case.

Video is via the i3 and motherboard.

The box is configured as a NAS and so no sound chipset is required.
Comment 1 Simon Lea 2011-06-15 03:53:25 UTC 
The Intel GT 1000/Pro specs;

Data Rate(s) Supported Per Port 10/100/1000 Mbps
Integrated Memory 64 KB
Interrupt Levels INTA
IEEE Support 802.2 and 802.3ab
Data Path Width 32-bit PCI
Hardware Certifications FCC B, CE, BSMI B, VCCI B
Data Transfer Mode Bus-master DMA
Controller Processor Intel® 82541PI
Typical Power Consumption 800 mA @+5 VDC
Comment 2 Chuck Ebbert 2011-06-27 06:19:42 UTC 
Please attach the output of the "dmesg" command and the output of "lspci -vnn" (separate plain-text attachments.) One of those four drivers in that list (the first is probably the SATA controller on the motherboard) is ignoring its interrupts, or maybe there is some other device issuing IRQ 16 that has no driver.

(You can sometimes work around the irqXX: nobody cared problem by adding "noirqdebug" to the kernel boot options, but that's just ignoring the real bug.)
Comment 3 Simon Lea 2011-06-27 12:02:13 UTC 
Hi Chuck,

Sorry but the machine is in active use at home so I had to rebuild it using an alternative OS for now.

I did capture a little bit more info which may answer your request though below.

Yeah, saw the message about ignoring but as you say, turning a blind eye doesn't fix the issue.

Thanks for taking a look.

Jun 15 20:45:15 nas kernel: [ 0.439609] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
Jun 15 20:45:15 nas kernel: [ 0.440900] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force'
Jun 15 20:45:15 nas kernel: [ 0.440904] pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jun 15 20:45:15 nas kernel: [ 0.441076] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force'
Jun 15 20:45:15 nas kernel: [ 0.441079] pci 0000:00:1c.0: PCI bridge to [bus 02-02]
Jun 15 20:45:15 nas kernel: [ 0.443111] pci 0000:00:1c.4: PCI bridge to [bus 03-03]
Jun 15 20:45:15 nas kernel: [ 0.445108] pci 0000:00:1c.5: PCI bridge to [bus 04-04]
Jun 15 20:45:15 nas kernel: [ 0.445278] pci 0000:00:1c.7: PCI bridge to [bus 05-06] (subtractive decode)
Jun 15 20:45:15 nas kernel: [ 0.445569] pci 0000:05:00.0: PCI bridge to [bus 06-06] (subtractive decode) 


::
::

Jun 15 20:45:15 nas kernel: [ 0.467163] pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jun 15 20:45:15 nas kernel: [ 0.467165] pci 0000:00:01.0: bridge window [io 0xe000-0xefff]
Jun 15 20:45:15 nas kernel: [ 0.467168] pci 0000:00:01.0: bridge window [mem 0xfb800000-0xfbafffff]
Jun 15 20:45:15 nas kernel: [ 0.467170] pci 0000:00:01.0: bridge window [mem pref disabled]
Jun 15 20:45:15 nas kernel: [ 0.467173] pci 0000:00:1c.0: PCI bridge to [bus 02-02]
Jun 15 20:45:15 nas kernel: [ 0.467175] pci 0000:00:1c.0: bridge window [io 0xd000-0xdfff]
Jun 15 20:45:15 nas kernel: [ 0.467179] pci 0000:00:1c.0: bridge window [mem 0xfb400000-0xfb6fffff]
Jun 15 20:45:15 nas kernel: [ 0.467182] pci 0000:00:1c.0: bridge window [mem pref disabled]
Jun 15 20:45:15 nas kernel: [ 0.467188] pci 0000:00:1c.4: PCI bridge to [bus 03-03]
Jun 15 20:45:15 nas kernel: [ 0.467190] pci 0000:00:1c.4: bridge window [io 0xc000-0xcfff]
Jun 15 20:45:15 nas kernel: [ 0.467194] pci 0000:00:1c.4: bridge window [mem 0xfbe00000-0xfbefffff]
Jun 15 20:45:15 nas kernel: [ 0.467198] pci 0000:00:1c.4: bridge window [mem pref disabled]
Jun 15 20:45:15 nas kernel: [ 0.467203] pci 0000:00:1c.5: PCI bridge to [bus 04-04]
Jun 15 20:45:15 nas kernel: [ 0.467204] pci 0000:00:1c.5: bridge window [io disabled]
Jun 15 20:45:15 nas kernel: [ 0.467208] pci 0000:00:1c.5: bridge window [mem 0xfbd00000-0xfbdfffff]
Jun 15 20:45:15 nas kernel: [ 0.467211] pci 0000:00:1c.5: bridge window [mem pref disabled]
Jun 15 20:45:15 nas kernel: [ 0.467217] pci 0000:05:00.0: PCI bridge to [bus 06-06]
Jun 15 20:45:15 nas kernel: [ 0.467220] pci 0000:05:00.0: bridge window [io 0xb000-0xbfff]
Jun 15 20:45:15 nas kernel: [ 0.467227] pci 0000:05:00.0: bridge window [mem 0xfbc00000-0xfbcfffff]
Jun 15 20:45:15 nas kernel: [ 0.467232] pci 0000:05:00.0: bridge window [mem pref disabled]
Jun 15 20:45:15 nas kernel: [ 0.467241] pci 0000:00:1c.7: PCI bridge to [bus 05-06]
Jun 15 20:45:15 nas kernel: [ 0.467244] pci 0000:00:1c.7: bridge window [io 0xb000-0xbfff]
Jun 15 20:45:15 nas kernel: [ 0.467248] pci 0000:00:1c.7: bridge window [mem 0xfbc00000-0xfbcfffff]
Jun 15 20:45:15 nas kernel: [ 0.467251] pci 0000:00:1c.7: bridge window [mem pref disabled]
Jun 15 20:45:15 nas kernel: [ 0.467268] pci 0000:00:01.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jun 15 20:45:15 nas kernel: [ 0.467280] pci 0000:00:1c.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Jun 15 20:45:15 nas kernel: [ 0.467290] pci 0000:00:1c.4: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Jun 15 20:45:15 nas kernel: [ 0.467300] pci 0000:00:1c.5: PCI INT B -> GSI 16 (level, low) -> IRQ 16
Jun 15 20:45:15 nas kernel: [ 0.467315] pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jun 15 20:45:15 nas kernel: [ 0.467324] pci 0000:05:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Comment 4 Simon Lea 2011-08-18 02:59:43 UTC 
Also now logged on the Linux Kernal Bugzilla as there seem to be a number of other reports of the same issue.

https://bugzilla.kernel.org/show_bug.cgi?id=41322

Other reports

38632 - https://bugzilla.kernel.org/show_bug.cgi?id=38632
35332 - https://bugzilla.kernel.org/show_bug.cgi?id=35332
34242 - https://bugzilla.kernel.org/show_bug.cgi?id=34242
35332 - https://bugzilla.kernel.org/show_bug.cgi?id=35332
32242 - https://bugzilla.kernel.org/show_bug.cgi?id=32242
39122 - https://bugzilla.kernel.org/show_bug.cgi?id=39122
Comment 5 Josh Boyer 2012-06-06 13:52:10 UTC 
Are you still having issues with this on the 2.6.43/3.3 kernels?
Comment 6 Josh Boyer 2012-07-11 17:49:09 UTC 
Fedora 15 has reached it's end of life as of June 26, 2012.  As a result, we will not be fixing any remaining bugs found in Fedora 15.

In the event that you have upgraded to a newer release and the bug you reported is still present, please reopen the bug and set the version field to the newest release you have encountered the issue with.  Before doing so, please ensure you are testing the latest kernel update in that release and attach any new and relevant information you may have gathered.

Thank you for taking the time to file a report.  We hope newer versions of Fedora suit your needs.