Description of problem: When inserting a PCIE card with a pcie bridge chip on board (multi function cards like Dual fibre HBA and Dual GB NIC combo card ) The newly inserted device is not completely installed. Version-Release number of selected component (if applicable): kernel-2.6.8-72 How reproducible: 100% Steps to Reproduce: 1. Boot a system 2. modprobe pciehp pciehp_debug=1 3. insert a card with IDT pcie bridge chip 4. Turn on the card (hit attention button or echo 1 > /sys/bus/pci/slot/<SLOT#>/power) 5. lspci -tv Actual results: lspci -tv shows new HEX bus number and device, but pcie port driver did not complete setup and there is missing information. In fact the hotplug code **appears to** never finish running and **appears** hung. pciehp handles power on and the code to enumerate pciebus and make HW available to kernel doesw ot complete. Expected results: pceihp handles power on. pcie port driver probes HW and properly brings up new HW that lspci can display all properties for and HW is available to kernel. Additional info:
the Capabilities and mem regions listed after the hotplug add of the HW is very different from those listed when HW is brought up at boot. See lspci attachments and diff.
this device that was hotplug added is missing a line that should display IO Behind Bridge. I/O behind bridge: 00009000-0000aff The IO behind bridge is present when the system boots and the card is present and we do an lspci. If there is no mem region configured for the bridge then that might be cause for device drivers failing / hanging when attemptind to bind to devices behind the bridge. 0d:00.0 PCI bridge: Integrated Device Technology, Inc. PES16T4 PCI Express Switch (rev 0d) (prog-if 00 [Normal decode]) Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Bus: primary=0d, secondary=0e, subordinate=10, sec-latency=0 Memory behind bridge: d3e00000-d3ffffff Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- Capabilities: [40] Express Upstream Port IRQ 0 Device: Supported: MaxPayload 2048 bytes, PhantFunc 0, ExtTag+ Device: Latency L0s <64ns, L1 <1us Device: AtnBtn- AtnInd- PwrInd- Device: SlotPowerLimit 25.000000 Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported- Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- Device: MaxPayload 128 bytes, MaxReadReq 128 bytes Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 0 Link: Latency L0s <2us, L1 <4us Link: ASPM Disabled CommClk- ExtSynch- Link: Speed 2.5Gb/s, Width x8 Capabilities: [c0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 PME-Enable- DSel=0 DScale=0 PME- Capabilities: [100] Advanced Error Reporting Capabilities: [200] Virtual Channel
Looking at the /proc/iomem You can see that at boot time the kernel / pci Driver walks the tree and allocated iomem properly for IO cards withbridge chips. You can see that after hotplug remove and insert of IDT bridge IOCARD, there is no mem region allocated. Here is diff of /proc/iomem (boot vs hotadd) [gf199321@shaolin PCIEHP_47]$ diff -u iomem_boot.txt iomem_hotplugadd.txt --- iomem_boot.txt 2009-05-10 07:08:48.971113000 -0400 +++ iomem_hotplugadd.txt 2009-05-10 07:08:48.975944000 -0400 @@ -37,27 +37,7 @@ cdbf8000-cdbfbfff : 0000:00:16.6 cdbfc000-cdbfffff : 0000:00:16.7 cdc00000-d3dfffff : PCI Bus #07 - d3c00000-d3dfffff : PCI Bus #08 - d3c00000-d3cfffff : PCI Bus #09 - d3cf8000-d3cfbfff : 0000:09:00.0 - d3cfc000-d3cfffff : 0000:09:00.1 - d3d00000-d3dfffff : PCI Bus #0a - d3d60000-d3d7ffff : 0000:0a:00.0 - d3d80000-d3d9ffff : 0000:0a:00.0 - d3dc0000-d3ddffff : 0000:0a:00.1 - d3de0000-d3dfffff : 0000:0a:00.1 d3e00000-d9ffffff : PCI Bus #0d - d9e00000-d9ffffff : PCI Bus #0e - d9e00000-d9efffff : PCI Bus #0f - d9efd000-d9efdfff : 0000:0f:00.0 - d9efe800-d9efe8ff : 0000:0f:00.0 - d9efec00-d9efecff : 0000:0f:00.1 - d9eff000-d9efffff : 0000:0f:00.1 - d9f00000-d9ffffff : PCI Bus #10 - d9f60000-d9f7ffff : 0000:10:00.0 - d9f80000-d9f9ffff : 0000:10:00.0 - d9fc0000-d9fdffff : 0000:10:00.1 - d9fe0000-d9ffffff : 0000:10:00.1 da000000-dfffffff : PCI Bus #13 e0000000-efffffff : reserved f4f00000-faefffff : PCI Bus #19 Here is tree view of these devices +-03.0-[0000:07-0c]----00.0-[0000:08-0a]--+-02.0-[0000:09]--+-00.0 QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA | | \-00.1 QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA | \-04.0-[0000:0a]--+-00.0 Intel Corporation 82571EB Gigabit Ethernet Controller | \-00.1 Intel Corporation 82571EB Gigabit Ethernet Controller +-05.0-[0000:0d-12]----00.0-[0000:0e-10]--+-06.0-[0000:0f]--+-00.0 Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter | | \-00.1 Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter | \-07.0-[0000:10]--+-00.0 Intel Corporation 82571EB Gigabit Ethernet Controller | \-00.1 Intel Corporation 82571EB Gigabit Ethernet Controller +-07.0-[0000:13-18]--
The *real issue * is that drivers/pci/hotplugpciehprm_acpi.c Devices that are hot added have IO, MEM, Busmastering and parity error disabled. The code that is supposed to enable such things is wrapped in an if statement and will NEVER be executed. When drivers come up and try to use the devices the lpfc and qla2400 hang the system while the e1000 does not hang but complains about invalid EEPROM. The call to pci_bus_write_config_word should be executed ALL of the time, not just in when command != cmd. After adding the code below to force the writes, hotplug added cards come up and have those bits set properly. Drivers load and IO works. The below is an example to prove the root cause and is not intended as the exact fix. Please review and provide any feedback. thanks void pciehprm_enable_card( struct controller *ctrl, struct pci_func *func, u8 card_type) ... ... ... - if (command != cmd) { - rc = pci_bus_write_config_word(pci_bus, devfn, PCI_COMMAND, command); - } + rc = pci_bus_write_config_word(pci_bus, devfn, PCI_COMMAND, command); - if ((card_type == PCI_HEADER_TYPE_BRIDGE) && (bcommand != bcmd)) { - rc = pci_bus_write_config_word(pci_bus, devfn, PCI_BRIDGE_CONTROL, bcommand); - } + if ((card_type == PCI_HEADER_TYPE_BRIDGE)) { + rc = pci_bus_write_config_word(pci_bus, devfn, PCI_BRIDGE_CONTROL, bcommand); + } }
In addition, The hotplug code is not like later versions found in 5.3 or fedora in that no calls to the kernel are made to allocate resources. this causes difference in /proc/iomem where the kernel would allocate the resources for the hotplugged card the same way as at boot time. the pciehp code manages this itself and collapses the IO space into one region in other words the kernel does cdc00000-d3dfffff : PCI Bus #07 d3c00000-d3dfffff : PCI Bus #08 d3c00000-d3cfffff : PCI Bus #09 d3cf8000-d3cfbfff : 0000:09:00.0 d3cfc000-d3cfffff : 0000:09:00.1 d3d00000-d3dfffff : PCI Bus #0a d3d60000-d3d7ffff : 0000:0a:00.0 d3d80000-d3d9ffff : 0000:0a:00.0 d3dc0000-d3ddffff : 0000:0a:00.1 d3de0000-d3dfffff : 0000:0a:00.1 AND PCIEHP does not really bring up subordinate BUses under the bridge and after removing and adding does this (which works but is less accurate, with no busses displayed) cdc00000-d3dfffff : PCI Bus #07 cdc00000-cdc03fff : qla2400 cdc04000-cdc07fff : qla2400 cdd00000-cdd1ffff : e1000 cdd20000-cdd3ffff : e1000 cdd40000-cdd5ffff : e1000 cdd60000-cdd7ffff : e1000 you can see /proc/iomem has lost pci bus 0x8, 0x9 and 0xa you can see also the kernel allocates mem regions from end to beginning and pcie does it from beginning to end. later versions of pciehp just make x function calls to the kernel rather than manage this stuff on its own, and you get a consistent allocation of resources at boot and at hotplug add.
In 2.6.15, commit a8a2be949267cb0d1d933a92d9fb43eda4f4fe88 changed the code to Reduce the PCI Express hotplug driver's dependence on ACPI. This eliminated pciehprm_enable_code (and its code).
Noticed additional information from the customer: After failed bringup of devices on pcieslot. subsequent use of pciehp action (attention button or sysfs file manipulation) on the slot will fail. reboot needed to clean up the slot and use HW. While in the state after an attempted HP add I installed kernel sources no prob Obtained latest 4.8 smp kernel Attempted to install 4.8 kernel. The Machine HUNG kernel is unstable after hotplug add and eventually becomes non responsive. Shot it with NMI ad no response