Bug 241257

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Robert, what BIOS version are you running?

P.

Adding Doug :)

P.

Tried this on my xw9300.  I see the following error:

ip_tables: (C) 2000-2002 Netfilter core team
e1000: eth0: e1000_request_irq: Unable to allocate MSI interrupt Error: -22
ADDRCONF(NETDEV_UP): eth0: link is not ready
e1000: eth0: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
eth0: no IPv6 routers present

Oddly, the interface is brought "up" in spite of the error ...

P.

My PCI layout is:

[root@dhcp83-55 ~]# lspci -t
-+-[0000:80]-+-00.0
 |           +-01.0
 |           \-0e.0-[0000:81]----00.0
 +-[0000:40]-+-01.0-[0000:41]--
 |           +-01.1
 |           +-02.0-[0000:61]--+-06.0
 |           |                 \-06.1
 |           \-02.1
 \-[0000:00]-+-00.0
             +-01.0
             +-01.1
             +-02.0
             +-02.1
             +-04.0
             +-06.0
             +-07.0
             +-08.0
             +-09.0-[0000:05]----05.0
             +-0a.0
             +-0e.0-[0000:0a]----00.0
             +-18.0
             +-18.1
             +-18.2
             +-18.3
             +-19.0
             +-19.1
             +-19.2
             \-19.3
[root@dhcp83-55 ~]# lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio
Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0a.0 Ethernet controller: nVidia Corporation CK804 Ethernet Controller (rev a3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM
Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM
Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
05:05.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000
Controller (PHY/Link)
0a:00.0 VGA compatible controller: nVidia Corporation NV44 [GeForce 6200
TurboCache(TM)] (rev a1)
40:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
40:01.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
40:02.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
40:02.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
61:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X
Fusion-MPT Dual Ultra320 SCSI (rev 07)
61:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X
Fusion-MPT Dual Ultra320 SCSI (rev 07)
80:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
80:01.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
80:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
81:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (rev 06)
[root@dhcp83-55 ~]# 

From dmesg,

PCI: Found HT MSI mapping on 0000:00:0e.0 with capability disabled

So why is the card coming up and requesting MSI?

P.

Ooops -- wrong cut-and-paste.

From dmesg,

PCI: MSI quirk detected. MSI disabled on chipset 0000:80:0e.0.

So why is the card coming up and requesting MSI?  The bridge has disabled MSI,
so this should fail.

P.

Oops .... looks like someone/something re-installed my system.  Ignore comments
5-8, which are from RHEL4.

P.

Okay, after finally getting onto the right kernel I've managed to quickly track
this down.  

We need the patchset beginning at message 2302 in the link below:

http://marc.info/?l=linux-kernel&m=115188796019151&w=2

P.

FYI, looks like there was a fix made in later kernel versions for this. With
vanilla 2.6.21 I get this output and MSI is disabled automatically:

PCI: Found disabled HT MSI Mapping on 0000:80:0e.0
PCI: Found disabled HT MSI Mapping on 0000:80:00.0
PCI: MSI quirk detected. MSI disabled on chipset 0000:80:0e.0.

This is from this code in drivers/pci/quirks.c:

/* The nVidia CK804 chipset may have 2 HT MSI mappings.
 * MSI are supported if the MSI capability set in any of these mappings.
 */
static void __devinit quirk_nvidia_ck804_msi_ht_cap(struct pci_dev *dev)
{
	struct pci_dev *pdev;

	if (!dev->subordinate)
		return;

	/* check HT MSI cap on this chipset and the root one.
	 * a single one having MSI is enough to be sure that MSI are supported.
	 */
	pdev = pci_get_slot(dev->bus, 0);
	if (!pdev)
		return;
	if (!msi_ht_cap_enabled(dev) && !msi_ht_cap_enabled(pdev)) {
		printk(KERN_WARNING "PCI: MSI quirk detected. "
		       "MSI disabled on chipset %s.\n",
		       pci_name(dev));
		dev->subordinate->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
	}
	pci_dev_put(pdev);
}
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, PCI_DEVICE_ID_NVIDIA_CK804_PCIE,
			quirk_nvidia_ck804_msi_ht_cap);

I guess the actual commits are to gregkh's pci-2.6 tree and were imported here:

http://marc.info/?l=linux-pci&m=115931884922956&w=2

P.

It looks like this can be resolved by the following MSI patches from grekh's
pci-2.6 tree:


commit 3f79e107f72e8efa86cd2f21356692b712713b5c
Author: Brice Goglin <brice>
Date:   Thu Aug 31 01:54:56 2006 -0400

    MSI: Cleanup existing MSI quirks
    
    Move MSI quirks in CONFIG_PCI_MSI, document why the serverworks quirk
    does not simply set PCI_BUS_FLAGS_NO_MSI, and create a generic quirk
    for other chipsets where setting PCI_BUS_FLAGS_NO_MSI is fine.

commit 24334a12533e9ac70dcb467ccd629f190afc5361
Author: Brice Goglin <brice>
Date:   Thu Aug 31 01:55:07 2006 -0400

    MSI: Factorize common code in pci_msi_supported()
    
    pci_enable_msi() and pci_enable_msix() use the same code to detect
    whether MSI might be enabled on this device. Factorize this code in
    pci_msi_supported(). And improve the documentation about the fact
    that only the root chipset must support MSI, but it is hard to
    find the root bus so we check all parent busses MSI flags.

commit fe97064c2870e174a6ff4a93feb11a70c4b71cc5
Author: Brice Goglin <brice>
Date:   Thu Aug 31 01:55:15 2006 -0400

    MSI: Export the PCI_BUS_FLAGS_NO_MSI flag in sysfs
    
    Export the PCI_BUS_FLAGS_NO_MSI flag of a PCI bus in the sysfs files
    of its parent device and make it writable. Could be used to:
    * disable MSI on a device which has not been blacklisted yet
    * allow MSI when some setpci hacks enable MSI support (for instance
      on the ServerWorks HT2000 chipset where the MSI HT cap is disabled
      by default).
    Architecture where some bus have no parent chipset cannot use this
    strategy to change MSI support.
    
    If the chipset does not have a subordinate bus, its 'bus_msi' file
    is empty.
    
    Also document and warn about the possible danger of changing the flag.

commit 46ff34633ed09f36ebc4b5c40ac37e592172df74
Author: Brice Goglin <brice>
Date:   Thu Aug 31 01:55:24 2006 -0400

    MSI: Rename PCI_CAP_ID_HT_IRQCONF into PCI_CAP_ID_HT
    
    0x08 is the HT capability, while PCI_CAP_ID_HT_IRQCONF would be
    the subtype 0x80 that mpic_scan_ht_pic() uses.
    Rename PCI_CAP_ID_HT_IRQCONF into PCI_CAP_ID_HT.
    
    And by the way, use it in the ipath driver instead of defining its
    own HT_CAPABILITY_ID.

commit 6397c75cbc4d7dbc3d07278b57c82a47dafb21b5
Author: Brice Goglin <brice>
Date:   Thu Aug 31 01:55:32 2006 -0400

    MSI: Blacklist PCI-E chipsets depending on Hypertransport MSI capability
    
    Introduce msi_ht_cap_enabled() to check the MSI capability in the
    Hypertransport configuration space.
    It is used in a generic quirk quirk_msi_ht_cap() to check whether
    MSI is enabled on hypertransport chipset, and a nVidia specific quirk
    quirk_nvidia_ck804_msi_ht_cap() where two 2 HT MSI mappings have to
    be checked.
    Both quirks set the PCI_BUS_FLAGS_NO_MSI bus flag when MSI is disabled.

I've ported them to RHEL5 and will attach them here shortly.  I've tested on the
xw9300 and it fixes the problems with the e1000.

P.

RHEL5 needs many MSI-related fixes.  I just posted several to an internal RH
kernel mailing list with the subject "[RHEL5 PATCH prelim] Update ATA, MSI, ICHx
quirks"

This backports what is currently in 2.6.22-rc4.

Thanks jgarzik -- your patchset fixes this issue.  I'm reassigning this to you
so that you can dup to the BZ of your pleasure :)

P.

*** Bug 234606 has been marked as a duplicate of this bug. ***

This request was previously evaluated by Red Hat Product Management
for inclusion in the current Red Hat Enterprise Linux release, but
Red Hat was unable to resolve it in time.  This request will be
reviewed for a future Red Hat Enterprise Linux release.

This request was previously evaluated by Red Hat Product Management
for inclusion in the current Red Hat Enterprise Linux release, but
Red Hat was unable to resolve it in time.  This request will be
reviewed for a future Red Hat Enterprise Linux release.

Appears to be fixed in Update 1 (2.6.18-53.1.6.el5):

PCI: Found disabled HT MSI Mapping on 0000:80:0e.0
PCI: Found disabled HT MSI Mapping on 0000:80:00.0
PCI: MSI quirk detected. MSI disabled on chipset 0000:80:0e.0.

These patches also disables MSI on DL385 G2 systems (MSI worked fine under
RHEL5, but is now disabled on RHEL5.1) and there seems to be no way to re-enable
it short of backing out the patch. It's disabled on the DL385 G2 due to the
HT1000 chipset being present as far as I can see.

This isn't too optimal for us because it kills the performance of 10 gbe.

There is a better solution to the HT1000 problem -- I'll see if I can dig that
up and give you some test kernels.  These 2 patches work that issue out:

commit 1d84b5424efbcce69a1c955ba181147d23d43a14
Author: David Miller <davem>
Date:   Thu Oct 25 01:15:53 2007 -0700

    PCI: Add MSI quirk for ServerWorks HT1000 PCIX bridge.

    This is the fix for the following problem:

    https://bugzilla.redhat.com/show_bug.cgi?id=227657

    The bnx2 device 5706 complains about MSI not working behind a
    ServerWorks HT1000 PCIX bridge. An earlier commit to fix the problem:

    e3008dedff4bdc96a5f67224cd3d8d12237082a0:

    "PCI: disable MSI by default on systems with Serverworks HT1000 chips"

    was not entirely correct, and has been reverted.

    MSI does not work on the PCIX bus because the BIOS did not set the
    HT_MSI_FLAGS_ENABLE bit in the HyperTransport MSI capability on the
    bridge.  We use the existing quirk_msi_ht_cap() to detect the problem
    and disable MSI in all buses behind it.

    Signed-off-by: Michael Chan <mchan>
    Cc: Anantha Subramanyam <ananth>
    Cc: Naren Sankar <nsankar>
    Signed-off-by: David S. Miller <davem>
    Acked-by: Jeff Garzik <jgarzik>
    Signed-off-by: Greg Kroah-Hartman <gregkh>

commit 2cc31879f8cfa0efc74fe7e58ab4e01ef5908730
Author: David Miller <davem>
Date:   Thu Oct 25 01:15:24 2007 -0700

    PCI: Revert "PCI: disable MSI by default on systems with Serverworks HT1000
chips"

    This reverts commit e3008dedff4bdc96a5f67224cd3d8d12237082a0.

    The real bug was an INTX issue in the tg3 ethernet chip, and
    cured by commit c129d962a66c76964954a98b38586ada82cf9381

    Signed-off-by: David S. Miller <davem>
    Acked-by: Jeff Garzik <jgarzik>
    Signed-off-by: Greg Kroah-Hartman <gregkh>

A single patch that will take care of the two upstream ones mentioned in comment
#29 is available here:

http://people.redhat.com/agospoda/rhel5/pci-disable-msi-on-ht1000-only.patch

Test kernels will be available soon and can be found here:

http://people.redhat.com/agospoda/#rhel5

Testing is always appreciated.

Too late to consider this patch for R5.2, moving out to R5.3.  

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Andy, I have a xw9300 in my cube.  I'll test using one of your test kernels.

P.

Andy, this seems to fix the problem on the xw9300.  I'm going to mark this a dup
of 438776.



*** This bug has been marked as a duplicate of 438776 ***