Bug 439110 - [Broadcom 5.3 bug] HT1000 chip based systems getting blacklisted for msi
Summary: [Broadcom 5.3 bug] HT1000 chip based systems getting blacklisted for msi
Keywords:
Status: CLOSED DUPLICATE of bug 438776
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Andy Gospodarek
QA Contact: Martin Jenner
URL:
Whiteboard:
Depends On: 439109
Blocks: 440221
TreeView+ depends on / blocked
 
Reported: 2008-03-27 01:17 UTC by Anantha Subramanyam
Modified: 2009-06-20 03:02 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2008-04-08 21:36:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
rhel5-test0.patch (2.50 KB, patch)
2008-03-27 13:16 UTC, Andy Gospodarek
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 43596 0 None None None Never

Description Anantha Subramanyam 2008-03-27 01:17:17 UTC
+++ This bug was initially created as a clone of Bug #439109 +++

Description of problem:
Following piece of code in quirks.c of RHEL5.1 code is disabling msi on all
HT1000 systems. This section of code has subsequently been refined in the later
kernels and those changes needs to be pulled into future releases.

===========================
/* Some chipsets do not support MSI. We cannot easily rely on setting
 * PCI_BUS_FLAGS_NO_MSI in its bus flags because there are actually
 * some other busses controlled by the chipset even if Linux is not
 * aware of it.  Instead of setting the flag on all busses in the
 * machine, simply disable MSI globally.
 */
static void __init quirk_disable_all_msi(struct pci_dev *dev)
{
	pci_no_msi();
	printk(KERN_WARNING "PCI: MSI quirk detected. MSI deactivated.\n");
}
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SERVERWORKS,
PCI_DEVICE_ID_SERVERWORKS_GCNB_LE, quirk_disable_all_msi);
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SERVERWORKS,
PCI_DEVICE_ID_SERVERWORKS_HT1000_PCIX, quirk_disable_all_msi);

==========================================================

Version-Release number of selected component (if applicable):


How reproducible:
Run RHEL5.1 on a HT1000 based platform

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 2 Andy Gospodarek 2008-03-27 13:16:11 UTC
Created attachment 299312 [details]
rhel5-test0.patch

This patch would probably resolve the issue.  

I just need to check to make sure our rhel5 tg3 driver is carrying the fixup
added upstream to address the original issue.

This patch:

commit 2fbe43f6f631dd7ce19fb1499d6164a5bdb34568
Author: Michael Chan <mchan>
Date:	Thu Sep 6 12:04:29 2007 +0100

    [TG3]: Workaround MSI bug on 5714/5780.

    A hardware bug was revealed after a recent PCI MSI patch was made to
    always disable legacy INTX when enabling MSI.  The 5714/5780 chips
    will not generate MSI when INTX is disabled, causing MSI failure
    messages to be reported, and another patch was made to workaround the
    problem by disabling MSI on ServerWorks HT1000 bridge chips commonly
    found with the 5714.

and this one:

commit ba698ad4b7e466cbb4a8bde6b9da8080ab06808d
Author: David Miller <davem>
Date:	Thu Oct 25 01:16:30 2007 -0700

    PCI: Add quirk for devices which disable MSI when INTX_DISABLE is set.

    A reasonably common problem with some devices is that they will
    disable MSI generation when the INTX_DISABLE bit is set in the
    PCI_COMMAND register.

    Quirk this explicitly, guarding the pci_intx() calls in msi.c with
    this quirk indication.

    The first entries for this quirk are for 5714 and 5780 Tigon3 chips,
    and thus we can remove the workaround code from the tg3.c driver.

Comment 3 Andy Gospodarek 2008-03-27 13:45:40 UTC
We have the first one patch listed in comment #2, so we should be fine.  The
second fix in comment #2 would be nice, but not necessary since it's a general
fix and I'm not sure it's kABI-safe since it changes the size of struct pci_dev.

Comment 4 Andy Gospodarek 2008-03-27 17:50:16 UTC
Anantha,

We can try and get this added to RHEL5.2, but we will need some help.  If I can
get test kernels to you by tomorrow morning (at the latest) can you verify them
for me?  I don't seem to have any HT1000 systems around anymore.

Thanks!

Comment 5 Anantha Subramanyam 2008-03-27 18:16:14 UTC
Yes we can verify them, thanks.

Comment 6 Andy Gospodarek 2008-03-27 21:45:55 UTC
My test kernels have been updated to include a patch for this bugzilla.

http://people.redhat.com/agospoda/#rhel5

Please test them and report back your results.

Comment 7 Michael Chan 2008-03-27 23:43:25 UTC
Joe T., please test this kernel on 5714 using MSI. You don't need to use an 
HT1000 system as long as the driver can successfully run in MSI mode.

Thanks.


Comment 8 Andy Gospodarek 2008-03-28 01:33:22 UTC
It would be great if someone (Joe T?) could at least boot an HT1000 system as
well just to be sure that MSI status will depend on BIOS/system configuration
rather than just blindly disabling MSI on the entire system.

Comment 9 Anantha Subramanyam 2008-03-28 01:36:27 UTC
I will be specifically testing on a HT1000 system.

Comment 11 Anantha Subramanyam 2008-03-28 23:01:50 UTC
Tested on a HT1000 system. Good news is that MSI is usable on the system.

Bad news is that MSI is being force enabled on HT1000 bridge and there is no way
for BIOS to control this. MSI capability block in the PXB is hardwired and it is
not possible for the BIOS to turn it off. 

Is it possible to either use the later version of HT1000 patch which enables /
disables MSI based on the BIOS settings, or if that is dependent on too many
other things just revert the HT1000 blacklist patch (back to RHEL5 state)


Comment 12 Andy Gospodarek 2008-03-31 12:28:49 UTC
Ok, I will look more closely at the upstream changes and backport as any of them as I can.

Comment 15 Peter Martuccelli 2008-03-31 21:23:47 UTC
I cannot justify a performance issue at this time for R5.2.  Moving out to R5.3.

Comment 16 Anantha Subramanyam 2008-03-31 21:43:11 UTC
I don't believe there is a performance issue. Note that this is not an issue of
using MSI behind HT1000 PXB bridge, but just that system which have HT1000 chip
are being blacklisted for MSI. So MSI is basically unusable on these systems. 

Also note that this blacklisting was not there in RHEL5 and we have not had any
complaints on performance from any of our customers.

thx

Comment 17 Andy Gospodarek 2008-03-31 21:57:26 UTC
Anantha, I did not have a chance to get a patch to you today, but I will shoot
for a new patch and test kernel tomorrow with hopefully all the upstream fixes
that are needed.

Comment 18 Andy Gospodarek 2008-04-01 21:43:25 UTC
Anantha,

I was looking at comment #11 again, but I really don't see any difference
between what is being included in 2.6.25 and what was in the patch I gave you. 
It looks like something was changed during the 2.6.24 development cycle though.
 Check this out:

commit 6bae1d96c6d7dde078994f6cb98235fd46f8736b
Author: Sebastien Dugue <sebastien.dugue>
Date:   Thu Dec 13 16:09:25 2007 -0800

    PCI: quirk: enable MSI Mapping on HT1000

    Add a quirk to enable the MSI mapping capability on HyperTransport bridges.

    Wire Broadcom's HT1000 to use the quirk.

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 2fbf60e..63357f0 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -1711,9 +1711,33 @@ static void __devinit quirk_msi_ht_cap(struct pci_dev *dev)
 }
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SERVERWORKS,
PCI_DEVICE_ID_SERVERWORKS_HT2000_PCIE,
                        quirk_msi_ht_cap);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SERVERWORKS,
-                       PCI_DEVICE_ID_SERVERWORKS_HT1000_PXB,
-                       quirk_msi_ht_cap);
+
+
+/*
+ *  Force enable MSI mapping capability on HT bridges
+ */
+static void __devinit quirk_msi_ht_cap_enable(struct pci_dev *dev)
+{
+       int pos, ttl = 48;
+
+       pos = pci_find_ht_capability(dev, HT_CAPTYPE_MSI_MAPPING);
+       while (pos && ttl--) {
+               u8 flags;
+
+               if (pci_read_config_byte(dev, pos + HT_MSI_FLAGS, &flags) == 0) {
+                       printk(KERN_INFO "PCI: Enabling HT MSI Mapping on %s\n",
+                              pci_name(dev));
+

This patch seems to drop the bits that you feel would be good for RHEL.  Michael
Chan apparently ACK'd it, so maybe we should ask him why it might be better to
do this than checking with the bios.

Comment 19 Andy Gospodarek 2008-04-01 21:48:33 UTC
Sorry to pin this on you, Michael, but I'm curious if you can provide some
insight as to why you ACK'd 6bae1d96c6d7dde078994f6cb98235fd46f8736b.  The
upstream discussion seemed pretty sparse.  Thanks!

Comment 20 Michael Chan 2008-04-01 22:21:20 UTC
I was initially CC'ed because I did an earlier patch to check HT1000 to see if 
we could use MSI or not.  Some older BIOS did not properly enable the HT-MSI 
mapping bit, causing BNX2 driver to complain about MSI not working.  This was 
the earlier patch which essentially would silence the BNX2 warning:

1d84b5424efbcce69a1c955ba181147d23d43a14

The patch that you mentioned above superseded this patch that I did.  To me, 
it seemed to be an improvement over my patch.  The new patch would allow users 
with older BIOS to use MSI and so I ACK'ed it.

It's possible that there may be yet another better way of doing this.

Comment 21 Andy Gospodarek 2008-04-02 12:25:15 UTC
Thanks for the update, Michael.  The bug reported (Anantha) was hoping we would
include the commit you mentioned in comment #20, but I questioned if that was
better or worse than the latest upstream.

I presumed the later patch was better since it allowed systems that did not
explicitly enable MSI in BIOS to still use MSI with the HT1000, which you have
confirmed.

Anantha, if you find out anything else from your investigation let me know.

Comment 23 Andy Gospodarek 2008-04-03 14:13:20 UTC
Anantha, I'm going to re-work my patch to the latest in 2.6.25 (the one that
does not check the possible BIOS status) and simply enables MSI on HT1000
devices.  I will not be much different from the one I gave you the other day.

We can work on a patch that will normally enable MSI, but on some specific
systems where you can disable MSI in the BIOS we can allow checking of HT
capabilities and disable if the BIOS has set it that way, but this needs to be
tested first and pushed upstream before we ship it in RHEL.

Off hand, Do you know of any specific systems that support disabling MSI in the
BIOS?  I might like to try and get my hands on one so I could test out some
patches in the future.




Comment 24 Andy Gospodarek 2008-04-03 18:15:04 UTC
OK, so the patch I would propose it no different from the one included in my
gtest.45 kernel.

Comment 25 Anantha Subramanyam 2008-04-04 17:21:49 UTC
Andy, the current patch is ok by us. If required, we can revisit the patch that
provides BIOS control later, thanks.


Note You need to log in before you can comment on or make changes to this bug.