Bug 500396 - pciehp -insertion of hotplug devices with pcie bridge chips does not complete
pciehp -insertion of hotplug devices with pcie bridge chips does not complete
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.7
All Linux
high Severity high
: rc
: 4.9
Assigned To: Prarit Bhargava
Red Hat Kernel QE team
:
Depends On:
Blocks: 512218
  Show dependency treegraph
 
Reported: 2009-05-12 10:57 EDT by Marc Milgram
Modified: 2010-09-29 08:58 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-09-29 08:58:09 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Marc Milgram 2009-05-12 10:57:34 EDT
Description of problem:
When inserting a PCIE card with a pcie bridge chip on board (multi function cards like Dual fibre HBA and Dual GB NIC combo card ) The newly inserted device is not completely installed.

Version-Release number of selected component (if applicable):
kernel-2.6.8-72

How reproducible:
100%


Steps to Reproduce:
1. Boot a system
2. modprobe  pciehp pciehp_debug=1
3. insert a card with IDT pcie bridge chip
4. Turn on the card  (hit attention button or echo 1 > /sys/bus/pci/slot/<SLOT#>/power)
5. lspci -tv
  
Actual results:
lspci -tv  shows new HEX bus number and device,  but pcie port driver did not complete setup and there is missing information.  In fact the hotplug code **appears to** never finish running and **appears** hung.

pciehp handles power on and the code to enumerate pciebus and make HW available to kernel doesw ot complete.

Expected results:
pceihp handles power on.  pcie port driver probes HW and properly brings up new HW  that lspci can display all properties for and HW is available to kernel.

Additional info:
Comment 2 Marc Milgram 2009-05-12 11:01:12 EDT
the Capabilities and mem regions listed after the hotplug add of the HW is very different from those listed when HW is brought up at boot.
See lspci attachments and diff.
Comment 3 Marc Milgram 2009-05-12 11:01:55 EDT
this device that was hotplug added is missing a line that should display IO Behind Bridge.

I/O behind bridge: 00009000-0000aff

The IO behind bridge is present when the system boots and the card is present and we do an lspci.

If there is no mem region configured for the bridge then that might be cause for device drivers failing / hanging when attemptind to bind to devices behind the bridge.




0d:00.0 PCI bridge: Integrated Device Technology, Inc. PES16T4 PCI Express Switch (rev 0d) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Bus: primary=0d, secondary=0e, subordinate=10, sec-latency=0
Memory behind bridge: d3e00000-d3ffffff
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Express Upstream Port IRQ 0
Device: Supported: MaxPayload 2048 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: SlotPowerLimit 25.000000
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 0
Link: Latency L0s <2us, L1 <4us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100] Advanced Error Reporting
Capabilities: [200] Virtual Channel
Comment 4 Marc Milgram 2009-05-12 11:02:35 EDT
Looking at the /proc/iomem  
You can see that at boot time the kernel / pci Driver walks the tree and allocated iomem properly for IO cards withbridge chips.
You can see that after hotplug remove and insert of IDT bridge IOCARD, there is no mem region allocated.

Here is diff of /proc/iomem  (boot vs hotadd)

[gf199321@shaolin PCIEHP_47]$ diff -u iomem_boot.txt iomem_hotplugadd.txt
--- iomem_boot.txt      2009-05-10 07:08:48.971113000 -0400
+++ iomem_hotplugadd.txt        2009-05-10 07:08:48.975944000 -0400
@@ -37,27 +37,7 @@
cdbf8000-cdbfbfff : 0000:00:16.6
cdbfc000-cdbfffff : 0000:00:16.7
cdc00000-d3dfffff : PCI Bus #07
-  d3c00000-d3dfffff : PCI Bus #08
-    d3c00000-d3cfffff : PCI Bus #09
-      d3cf8000-d3cfbfff : 0000:09:00.0
-      d3cfc000-d3cfffff : 0000:09:00.1
-    d3d00000-d3dfffff : PCI Bus #0a
-      d3d60000-d3d7ffff : 0000:0a:00.0
-      d3d80000-d3d9ffff : 0000:0a:00.0
-      d3dc0000-d3ddffff : 0000:0a:00.1
-      d3de0000-d3dfffff : 0000:0a:00.1
d3e00000-d9ffffff : PCI Bus #0d
-  d9e00000-d9ffffff : PCI Bus #0e
-    d9e00000-d9efffff : PCI Bus #0f
-      d9efd000-d9efdfff : 0000:0f:00.0
-      d9efe800-d9efe8ff : 0000:0f:00.0
-      d9efec00-d9efecff : 0000:0f:00.1
-      d9eff000-d9efffff : 0000:0f:00.1
-    d9f00000-d9ffffff : PCI Bus #10
-      d9f60000-d9f7ffff : 0000:10:00.0
-      d9f80000-d9f9ffff : 0000:10:00.0
-      d9fc0000-d9fdffff : 0000:10:00.1
-      d9fe0000-d9ffffff : 0000:10:00.1
da000000-dfffffff : PCI Bus #13
e0000000-efffffff : reserved
f4f00000-faefffff : PCI Bus #19

Here is tree view of these devices

          +-03.0-[0000:07-0c]----00.0-[0000:08-0a]--+-02.0-[0000:09]--+-00.0  QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA
          |                                         |                 \-00.1  QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA
          |                                         \-04.0-[0000:0a]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet Controller
          |                                                           \-00.1  Intel Corporation 82571EB Gigabit Ethernet Controller
          +-05.0-[0000:0d-12]----00.0-[0000:0e-10]--+-06.0-[0000:0f]--+-00.0  Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter
          |                                         |                 \-00.1  Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter
          |                                         \-07.0-[0000:10]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet Controller
          |                                                           \-00.1  Intel Corporation 82571EB Gigabit Ethernet Controller
          +-07.0-[0000:13-18]--
Comment 5 Marc Milgram 2009-05-12 11:05:22 EDT
The *real issue * is that  drivers/pci/hotplugpciehprm_acpi.c
Devices that are hot added have IO, MEM, Busmastering and parity error disabled.
The code that is supposed to enable such things is wrapped in an if statement
and will NEVER be executed.
When drivers come up and try to use the devices  the lpfc and qla2400  hang the system while the e1000 does not hang but complains about invalid EEPROM.


The call to pci_bus_write_config_word   should be executed ALL of the time, not just in when command != cmd.
After adding the code below to force the writes,  hotplug added cards come up and have those bits set properly.  Drivers load and IO works.

The below is an example to prove the root cause and is not intended as the exact fix. Please review and provide any feedback.

thanks

void pciehprm_enable_card(
       struct controller *ctrl,
       struct pci_func *func,
       u8 card_type)
...
...
...

-       if (command != cmd) {
-               rc = pci_bus_write_config_word(pci_bus, devfn, PCI_COMMAND, command);
-       }
+       rc = pci_bus_write_config_word(pci_bus, devfn, PCI_COMMAND, command);

-       if ((card_type == PCI_HEADER_TYPE_BRIDGE) && (bcommand != bcmd)) {
-               rc = pci_bus_write_config_word(pci_bus, devfn, PCI_BRIDGE_CONTROL, bcommand);
-       }
+       if ((card_type == PCI_HEADER_TYPE_BRIDGE)) {
+               rc = pci_bus_write_config_word(pci_bus, devfn, PCI_BRIDGE_CONTROL, bcommand);
+       }
}
Comment 6 Marc Milgram 2009-05-12 11:05:50 EDT
In addition,
The hotplug code is not like later versions found in 5.3 or fedora in that no calls to the kernel are made to allocate resources.

this causes  difference in /proc/iomem  where
the kernel would allocate the resources for the hotplugged card the same way as at boot time.

the pciehp code manages this itself and collapses the IO space into one region

in other words the kernel does
cdc00000-d3dfffff : PCI Bus #07
 d3c00000-d3dfffff : PCI Bus #08
   d3c00000-d3cfffff : PCI Bus #09
     d3cf8000-d3cfbfff : 0000:09:00.0
     d3cfc000-d3cfffff : 0000:09:00.1
   d3d00000-d3dfffff : PCI Bus #0a
     d3d60000-d3d7ffff : 0000:0a:00.0
     d3d80000-d3d9ffff : 0000:0a:00.0
     d3dc0000-d3ddffff : 0000:0a:00.1
     d3de0000-d3dfffff : 0000:0a:00.1

AND PCIEHP does not really bring up subordinate BUses under the bridge
and after removing and adding  does this  (which works but is less accurate, with no busses displayed)

cdc00000-d3dfffff : PCI Bus #07
 cdc00000-cdc03fff : qla2400
 cdc04000-cdc07fff : qla2400
 cdd00000-cdd1ffff : e1000
 cdd20000-cdd3ffff : e1000
 cdd40000-cdd5ffff : e1000
 cdd60000-cdd7ffff : e1000

you can see /proc/iomem has lost pci bus 0x8, 0x9 and 0xa
you can see also the kernel allocates mem regions from end to beginning
and pcie does it from beginning to end.

later versions of pciehp just make x function calls to the kernel rather than manage this stuff on its own, and you get a consistent allocation of resources at boot and at hotplug add.
Comment 7 Marc Milgram 2009-05-12 11:09:09 EDT
In 2.6.15, commit a8a2be949267cb0d1d933a92d9fb43eda4f4fe88 changed the code to Reduce the PCI Express hotplug driver's dependence on ACPI.  This eliminated pciehprm_enable_code (and its code).
Comment 9 Marc Milgram 2009-05-12 11:16:42 EDT
Noticed additional information from the customer:

After failed bringup of devices on pcieslot.  subsequent use of pciehp action (attention button or sysfs file manipulation) on the slot will fail.

reboot needed to clean up the slot and use HW. 



While in the state after an attempted HP add
I installed kernel sources  no prob
Obtained latest 4.8 smp kernel
Attempted to install 4.8 kernel.
The Machine HUNG

kernel is unstable after hotplug add and eventually becomes non responsive.
Shot it with NMI ad no response

Note You need to log in before you can comment on or make changes to this bug.