Bug 1467674 - Kernel driver (ethernet) crash when PCIe Extended Tags are enabled on Tigon3 ethernet
Kernel driver (ethernet) crash when PCIe Extended Tags are enabled on Tigon3 ...
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: kerneloops (Show other bugs)
25
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Orphan Owner
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-04 10:06 EDT by Wim ten Have
Modified: 2017-12-12 05:26 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-12 05:26:36 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Wim ten Have 2017-07-04 10:06:19 EDT
Description of problem:
=======================
Systems with "eth0: Tigon3 [partno(BCM95721) rev 4201] (PCI Express)" ethernet like DELL PowerEdge SC1435 fail their ethernet after interface bind/ifconfig up.
  [    0.000000] SMBIOS 2.4 present.
  [    0.000000] DMI: Dell Inc. PowerEdge SC1435/0H313M, BIOS 2.2.5 03/21/2008


The problem is not specific to this piece of h/w.  I did pin-point the issue to specific kernel code commit 60db3a4d8cc9073cf56264785197ba75ee1caca4
  * <wtenhave@hagen:55> git bisect good
    60db3a4d8cc9073cf56264785197ba75ee1caca4 is the first bad commit
    commit 60db3a4d8cc9073cf56264785197ba75ee1caca4
    Author: Sinan Kaya <okaya@codeaurora.org>
    Date:   Fri Jan 20 09:16:51 2017 -0500

      PCI: Enable PCIe Extended Tags if supported

The system will shortly after getting to the ifconfig up statement report below kernel messages.
  Jul  4 15:00:12 hagen kernel: tg3 0000:01:00.0 eth0: Tigon3 [partno(BCM95721) rev 4201] (PCI Express) MAC address 00:22:19:27:cd:f8
  Jul  4 15:00:12 hagen kernel: tg3 0000:01:00.0 eth0: attached PHY is 5750 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
  Jul  4 15:00:12 hagen kernel: tg3 0000:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
  Jul  4 15:00:12 hagen kernel: tg3 0000:01:00.0 eth0: dma_rwctrl[76180000] dma_mask[64-bit]
  Jul  4 15:00:12 hagen kernel: tg3 0000:02:00.0 eth1: Tigon3 [partno(BCM95721) rev 4201] (PCI Express) MAC address 00:22:19:27:cd:f9
  Jul  4 15:00:12 hagen kernel: tg3 0000:02:00.0 eth1: attached PHY is 5750 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
  Jul  4 15:00:12 hagen kernel: tg3 0000:02:00.0 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
  Jul  4 15:00:12 hagen kernel: tg3 0000:02:00.0 eth1: dma_rwctrl[76180000] dma_mask[64-bit]
     ...
  Jul  4 15:00:12 hagen kernel: tg3 0000:02:00.0 enp2s0: renamed from eth1
     ...
  Jul  4 15:00:39 hagen kernel: tg3 0000:01:00.0 enp1s0: Link is up at 1000 Mbps, full duplex
  Jul  4 15:00:39 hagen kernel: tg3 0000:01:00.0 enp1s0: Flow control is on for TX and on for RX
     ...
  Jul  4 15:00:50 hagen kernel: NETDEV WATCHDOG: enp1s0 (tg3): transmit queue 0 timed out
  Jul  4 15:00:50 hagen kernel: ------------[ cut here ]------------
  Jul  4 15:00:50 hagen kernel: WARNING: CPU: 6 PID: 0 at net/sched/sch_generic.c:316 dev_watchdog+0x215/0x220
  Jul  4 15:00:50 hagen kernel: Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_raw ip6table_security ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle iptable_raw iptable_security iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables sunrpc xfs libcrc32c amd64_edac_mod edac_mce_amd kvm_amd kvm dcdbas irqbypass ipmi_ssif acpi_cpufreq shpchp ipmi_si ipmi_devintf tpm_tis pcspkr tpm_tis_core k10temp i2c_piix4 ipmi_msghandler tpm target_core_mod amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper ttm drm tg3 ptp ata_generic serio_raw pata_acpi pps_core pata_serverworks sata_svw
  Jul  4 15:00:50 hagen kernel: CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.12.0broken+ #16
  Jul  4 15:00:50 hagen kernel: Hardware name: Dell Inc. PowerEdge SC1435/0H313M, BIOS 2.2.5 03/21/2008
  Jul  4 15:00:50 hagen kernel: task: ffff90a9ede5c980 task.stack: ffffb123031b4000
  Jul  4 15:00:50 hagen kernel: RIP: 0010:dev_watchdog+0x215/0x220
  Jul  4 15:00:50 hagen kernel: RSP: 0018:ffff90a9efd83e60 EFLAGS: 00010286
  Jul  4 15:00:50 hagen kernel: RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
  Jul  4 15:00:50 hagen kernel: RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 0000000000000300
  Jul  4 15:00:50 hagen kernel: RBP: ffff90a9efd83e80 R08: 0000000000000000 R09: 0000000000000346
  Jul  4 15:00:50 hagen kernel: R10: ffff90a9efd92430 R11: 000000000000000f R12: ffff90a9ece46000
  Jul  4 15:00:50 hagen kernel: R13: 0000000000000006 R14: 0000000000000005 R15: ffff90a9ece46000
  Jul  4 15:00:50 hagen kernel: FS:  0000000000000000(0000) GS:ffff90a9efd80000(0000) knlGS:0000000000000000
  Jul  4 15:00:50 hagen kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  Jul  4 15:00:50 hagen kernel: CR2: 000055d4fa41f038 CR3: 00000004c3e09000 CR4: 00000000000006e0
  Jul  4 15:00:50 hagen kernel: Call Trace:
  Jul  4 15:00:50 hagen kernel: <IRQ>
  Jul  4 15:00:50 hagen kernel: ? qdisc_rcu_free+0x50/0x50
  Jul  4 15:00:50 hagen kernel: call_timer_fn+0x35/0x130
  Jul  4 15:00:50 hagen kernel: run_timer_softirq+0x1d1/0x420
  Jul  4 15:00:50 hagen kernel: ? sched_clock+0x9/0x10
  Jul  4 15:00:50 hagen kernel: ? sched_clock+0x9/0x10
  Jul  4 15:00:50 hagen kernel: ? sched_clock_cpu+0x11/0xb0
  Jul  4 15:00:50 hagen kernel: __do_softirq+0x10c/0x2a5
  Jul  4 15:00:50 hagen kernel: irq_exit+0xff/0x110
  Jul  4 15:00:50 hagen kernel: smp_apic_timer_interrupt+0x3d/0x50
  Jul  4 15:00:50 hagen kernel: apic_timer_interrupt+0x93/0xa0
  Jul  4 15:00:50 hagen kernel: RIP: 0010:native_safe_halt+0x6/0x10
  Jul  4 15:00:50 hagen kernel: RSP: 0018:ffffb123031b7e60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
  Jul  4 15:00:50 hagen kernel: RAX: 6874754100002548 RBX: ffff90a9ede5c980 RCX: 0000000000000000
  Jul  4 15:00:50 hagen kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
  Jul  4 15:00:50 hagen kernel: RBP: ffffb123031b7e60 R08: 00000009c80ff365 R09: ffffb1230837fa38
  Jul  4 15:00:50 hagen kernel: R10: 0000000000000000 R11: 00000000fffc0fe9 R12: 0000000000000006
  Jul  4 15:00:50 hagen kernel: R13: ffff90a9ede5c980 R14: 0000000000000000 R15: 0000000000000000
  Jul  4 15:00:50 hagen kernel: </IRQ>
  Jul  4 15:00:50 hagen kernel: default_idle+0x20/0x100
  Jul  4 15:00:50 hagen kernel: amd_e400_idle+0x3f/0x50
  Jul  4 15:00:50 hagen kernel: arch_cpu_idle+0xf/0x20
  Jul  4 15:00:50 hagen kernel: default_idle_call+0x23/0x30
  Jul  4 15:00:50 hagen kernel: do_idle+0x174/0x1e0
  Jul  4 15:00:50 hagen kernel: cpu_startup_entry+0x71/0x80
  Jul  4 15:00:50 hagen kernel: start_secondary+0x154/0x190
  Jul  4 15:00:50 hagen kernel: secondary_startup_64+0x9f/0x9f
  Jul  4 15:00:50 hagen kernel: Code: 8c 24 64 04 00 00 eb 8f 4c 89 e7 c6 05 ef 0e 88 00 01 e8 4f 6b fd ff 89 d9 48 89 c2 4c 89 e6 48 c7 c7 60 a0 d1 86 e8 12 cc a5 ff <0f> ff eb c1 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 c7 47 08 00 
  Jul  4 15:00:50 hagen kernel: ---[ end trace 6fdc4540cb931145 ]---
  Jul  4 15:00:50 hagen kernel: tg3 0000:01:00.0 enp1s0: transmit timed out, resetting
  Jul  4 15:00:52 hagen abrt-dump-journal-oops: abrt-dump-journal-oops: Found oopses: 1
  Jul  4 15:00:52 hagen abrt-dump-journal-oops: abrt-dump-journal-oops: Creating problem directories
  Jul  4 15:00:52 hagen kernel: tg3 0000:01:00.0 enp1s0: 0x00000000: 0x165914e4, 0x00100406, 0x02000021, 0x00000010
  Jul  4 15:00:52 hagen kernel: tg3 0000:01:00.0 enp1s0: 0x00000010: 0xefef0004, 0x00000000, 0x00000000, 0x00000000
    ...
  Jul  4 15:00:53 hagen kernel: tg3 0000:01:00.0 enp1s0: 0x00007810: 0x00000000, 0x00000060, 0x00000000, 0x00000000
  Jul  4 15:00:53 hagen kernel: tg3 0000:01:00.0 enp1s0: 0: Host status block [00000001:00000014:(0000:0005:0000):(0005:000c)]
  Jul  4 15:00:53 hagen kernel: tg3 0000:01:00.0 enp1s0: 0: NAPI info [00000014:00000014:(0010:000c:01ff):0005:(00cd:0000:0000:0000)]
  Jul  4 15:00:53 hagen kernel: tg3 0000:01:00.0: tg3_stop_block timed out, ofs=4800 enable_bit=2
  Jul  4 15:00:53 hagen kernel: tg3 0000:01:00.0 enp1s0: Link is down
  Jul  4 15:00:53 hagen abrt-dump-journal-oops: Reported 1 kernel oopses to Abrt
  Jul  4 15:00:53 hagen abrt-server: Deleting problem directory oops-2017-07-04-15:00:52-889-0 (dup of oops-2017-07-03-12:49:03-930-0)


Version-Release number of selected component (if applicable):
=============================================================
The issue was first noticeable under Fedora 25 updating the kernel from version 4.10.x => 4.11.0.  

Given I needed to move forward with latest version of available kernel i decided to hunt the bug and report.  
I found it to be cause under all (linux) kernels from commit 60db3a4d8cc9073cf56264785197ba75ee1caca4

* <wtenhave@hagen:55> git bisect good
  60db3a4d8cc9073cf56264785197ba75ee1caca4 is the first bad commit
  commit 60db3a4d8cc9073cf56264785197ba75ee1caca4
  Author: Sinan Kaya <okaya@codeaurora.org>
  Date:   Fri Jan 20 09:16:51 2017 -0500

    PCI: Enable PCIe Extended Tags if supported



How reproducible:
=================
It is 100% reproducible.  In fact you can take latest kernel out today and back-out the change as done under;
  commit 60db3a4d8cc9073cf56264785197ba75ee1caca4

  <wtenhave@hagen:55> git log 60db3a4d8cc9073cf56264785197ba75ee1caca4
  commit 60db3a4d8cc9073cf56264785197ba75ee1caca4
  Author: Sinan Kaya <okaya@codeaurora.org>
  Date:   Fri Jan 20 09:16:51 2017 -0500

    PCI: Enable PCIe Extended Tags if supported
    
    Every PCIe device can generate 5-bit transaction Tags, which allow up to 32
    concurrent requests.  Some devices can generate 8-bit Extended Tags, which
    allow up to 256 concurrent requests.
    
    Per the ECN mentioned below, all PCIe Receivers are expected to support
    Extended Tags, so devices are allowed (but not required) to enable them by
    default.
    
    If a device supports Extended Tags but does not enable them by default,
    enable them.  This allows the device to have up to 256 outstanding
    transactions at a time, which may improve performance.
    
    [bhelgaas: changelog, check for PCIe device]
    Link: https://pcisig.com/sites/default/files/specification_documents/ECN_Extended_Tag_Enable_Default_05Sept2008_final.pdf
    Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>


To back-out take any kernel later then 4.11 and apply below code change.  Then build and install that kernel.

  <wtenhave@hagen:58> git diff
  diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
  index 19c8950..1005e9d 100644
  --- a/drivers/pci/probe.c
  +++ b/drivers/pci/probe.c
  @@ -1707,7 +1707,7 @@ static void pci_configure_device(struct pci_dev *dev)
          int ret;
   
          pci_configure_mps(dev);
  -       pci_configure_extended_tags(dev);
  +       // pci_configure_extended_tags(dev);
   
          memset(&hpp, 0, sizeof(hpp));
          ret = pci_get_hp_params(dev, &hpp);


Steps to Reproduce:
===================
1. Take a machine with appropriate h/w like a Dell Inc. PowerEdge SC1435/0H313M, BIOS 2.2.5 03/21/2008 with Tigon3 [partno(BCM95721) rev 4201] (PCI Express) controller
2. Install Fedora25 or (other) with kernel including specific code/commit like kernel-4.11.7-200.fc25.x86_64
3. Boot and see it crash as soon it starts to operate on specific PCI Express Ethernet controller.

Actual results:

Expected results:

Additional info:
================
If you need further input please drop me a line at;  "Wim ten Have <wim.ten.have@oracle.com>"
Comment 1 Wim ten Have 2017-07-04 13:50:31 EDT
Additional information added per developers request:
====================================================
<wtenhave@nina:52> ssh hagen
Last login: Tue Jul  4 19:47:03 2017 from 192.168.1.22

<wtenhave@hagen:52> uname -a
Linux hagen 4.12.0+ #15 SMP Tue Jul 4 13:38:06 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux

<wtenhave@hagen:53> sudo lspci -vvv
00:01.0 PCI bridge: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	NUMA node: 0
	Bus: primary=00, secondary=03, subordinate=04, sec-latency=64
	I/O behind bridge: 0000e000-0000efff [size=4K]
	Memory behind bridge: efc00000-efdfffff [size=2M]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [empty]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [90] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
		Status: Dev=00:01.0 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=0 CommitmentLimit=0
		Downstream: Capacity=0 CommitmentLimit=0
	Capabilities: [a0] HyperTransport: MSI Mapping Enable+ Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [b0] HyperTransport: Slave or Primary Interface
		Command: BaseUnitID=1 UnitCnt=3 MastHost- DefDir-
		Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0
		Link Config 0: MLWI=8bit MLWO=8bit LWI=8bit LWO=8bit
		Link Control 1: CFlE+ CST- CFE- <LkFail- Init- EOC+ TXO- <CRCErr=0
		Link Config 1: MLWI=8bit MLWO=8bit LWI=8bit LWO=8bit
		Revision ID: 0.00
	Capabilities: [d8] Subsystem: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge
	Kernel modules: shpchp

00:02.0 Host bridge: Broadcom BCM5785 [HT1000] Legacy South Bridge
	Subsystem: Broadcom Device 0201
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	NUMA node: 0
	Kernel driver in use: piix4_smbus
	Kernel modules: i2c_piix4

00:02.1 IDE interface: Broadcom BCM5785 [HT1000] IDE (prog-if 8a [Master SecP PriP])
	Subsystem: Dell Device 01eb
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 64 bytes
	NUMA node: 0
	Region 0: I/O ports at 01f0 [size=8]
	Region 1: I/O ports at 03f4
	Region 2: I/O ports at 0170 [size=8]
	Region 3: I/O ports at 0374
	Region 4: I/O ports at 08c0 [size=16]
	Capabilities: [b0] #00 [0000]
	Kernel driver in use: pata_serverworks
	Kernel modules: pata_serverworks, pata_acpi, ata_generic

00:02.2 ISA bridge: Broadcom BCM5785 [HT1000] LPC
	Subsystem: Dell Device 01eb
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	NUMA node: 0

00:03.0 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01) (prog-if 10 [OHCI])
	Subsystem: Dell Device 01eb
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	NUMA node: 0
	Region 0: Memory at efbed000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at d000 [size=256]
	Capabilities: [dc] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Kernel driver in use: ohci-pci

00:03.1 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01) (prog-if 10 [OHCI])
	Subsystem: Dell Device 01eb
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	NUMA node: 0
	Region 0: Memory at efbee000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at d400 [size=256]
	Capabilities: [dc] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Kernel driver in use: ohci-pci

00:03.2 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01) (prog-if 20 [EHCI])
	Subsystem: Dell Device 01eb
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	NUMA node: 0
	Region 0: Memory at efbef000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at d800 [size=256]
	Capabilities: [dc] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Kernel driver in use: ehci-pci

00:04.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller])
	Subsystem: Dell Device 01eb
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64 (2000ns min), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 26
	NUMA node: 0
	Region 0: Memory at e0000000 (32-bit, prefetchable) [size=128M]
	Region 1: I/O ports at dc00 [size=256]
	Region 2: Memory at efbf0000 (32-bit, non-prefetchable) [size=64K]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: radeon
	Kernel modules: radeon

00:07.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 17
	NUMA node: 0
	Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [empty]
	Memory behind bridge: fff00000-000fffff [empty]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [empty]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [a0] HyperTransport: MSI Mapping Enable+ Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [b0] Express (v1) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled+ CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [98] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable+ Count=1/2 Maskable- 64bit+
		Address: 00000000fee5500c  Data: 4151
	Capabilities: [78] Subsystem: Device 0000:0000
	Capabilities: [50] HyperTransport: Slave or Primary Interface
		Command: BaseUnitID=7 UnitCnt=5 MastHost- DefDir- DUL-
		Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
		Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Link Control 1: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
		Link Config 1: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=8bit DwFcInEn- LWO=8bit DwFcOutEn-
		Revision ID: 2.00
		Link Frequency 0: 1.0GHz
		Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability 0: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC- LDTSTOP+ CRCTM- ECTLT- 64bA- UIDRD-
		Link Frequency 1: 1.0GHz
		Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability 1: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE- SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
		Prefetchable memory behind bridge Upper: 00-00
		Bus Number: 00
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [200 v1] Vendor Specific Information: ID=0001 Rev=1 Len=100 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:08.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 19
	NUMA node: 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [empty]
	Memory behind bridge: efe00000-efefffff [size=1M]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [empty]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [a0] HyperTransport: MSI Mapping Enable+ Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [b0] Express (v1) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [98] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable+ Count=1/2 Maskable- 64bit+
		Address: 00000000fee5500c  Data: 4171
	Capabilities: [78] Subsystem: Device 0000:0000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [200 v1] Vendor Specific Information: ID=0001 Rev=1 Len=100 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:09.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 21
	NUMA node: 0
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [empty]
	Memory behind bridge: eff00000-efffffff [size=1M]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [empty]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [a0] HyperTransport: MSI Mapping Enable+ Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [b0] Express (v1) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #3, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [98] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable+ Count=1/2 Maskable- 64bit+
		Address: 00000000fee5500c  Data: 4191
	Capabilities: [78] Subsystem: Device 0000:0000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [200 v1] Vendor Specific Information: ID=0001 Rev=1 Len=100 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:0a.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 23
	NUMA node: 0
	Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [empty]
	Memory behind bridge: fff00000-000fffff [empty]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [empty]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [a0] HyperTransport: MSI Mapping Enable+ Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [b0] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 2.5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled+ CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #1, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [98] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable+ Count=1/2 Maskable- 64bit+
		Address: 00000000fee5500c  Data: 41b1
	Capabilities: [78] Subsystem: Device 0000:0000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [200 v1] Vendor Specific Information: ID=0001 Rev=1 Len=100 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:0b.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 25
	NUMA node: 0
	Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [empty]
	Memory behind bridge: fff00000-000fffff [empty]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [empty]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [a0] HyperTransport: MSI Mapping Enable+ Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [b0] Express (v1) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #5, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled+ CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [98] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable+ Count=1/2 Maskable- 64bit+
		Address: 00000000fee5500c  Data: 41d1
	Capabilities: [78] Subsystem: Device 0000:0000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [200 v1] Vendor Specific Information: ID=0001 Rev=1 Len=100 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	Capabilities: [80] HyperTransport: Host or Secondary Interface
		Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+ Slave- <EOCErr- DUL-
		Link Control: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
		Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Revision ID: 3.00
		Link Frequency: 1.0GHz
		Link Error: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- ExtRS- UCnfE-
	Capabilities: [a0] HyperTransport: Host or Secondary Interface
		Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+ Slave- <EOCErr- DUL-
		Link Control: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
		Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Revision ID: 3.00
		Link Frequency: 1.0GHz
		Link Error: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- ExtRS- UCnfE-
	Capabilities: [c0] HyperTransport: Host or Secondary Interface
		Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+ Slave- <EOCErr- DUL-
		Link Control: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
		Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Revision ID: 3.00
		Link Frequency: 1.0GHz
		Link Error: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- ExtRS- UCnfE-

00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0

00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0

00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	Capabilities: [f0] Secure device <?>
	Kernel modules: k10temp

00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0

00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 1
	Capabilities: [80] HyperTransport: Host or Secondary Interface
		Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+ Slave- <EOCErr- DUL-
		Link Control: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
		Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Revision ID: 3.00
		Link Frequency: 1.0GHz
		Link Error: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- ExtRS- UCnfE-
	Capabilities: [a0] HyperTransport: Host or Secondary Interface
		Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+ Slave- <EOCErr- DUL-
		Link Control: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
		Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Revision ID: 3.00
		Link Frequency: 1.0GHz
		Link Error: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- ExtRS- UCnfE-
	Capabilities: [c0] HyperTransport: Host or Secondary Interface
		Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+ Slave- <EOCErr- DUL-
		Link Control: CFlE+ CST- CFE- <LkFail- Init- EOC+ TXO+ <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
		Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=N/C DwFcInEn- LWO=N/C DwFcOutEn-
		Revision ID: 3.00
		Link Frequency: 200MHz
		Link Error: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- ExtRS- UCnfE-

00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 1

00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 1

00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 1
	Capabilities: [f0] Secure device <?>
	Kernel modules: k10temp

00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 1

01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 21)
	Subsystem: Dell Device 01eb
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 27
	NUMA node: 0
	Region 0: Memory at efef0000 (64-bit, non-prefetchable) [size=64K]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] Vital Product Data
		Product Name: Broadcom NetXtreme Gigabit Ethernet Controller
		Read-only fields:
			[PN] Part number: BCM95721
			[EC] Engineering changes: 106679-15
			[SN] Serial number: 0123456789
			[MN] Manufacture ID: 31 34 65 34
			[RV] Reserved: checksum good, 28 byte(s) reserved
		Read/write fields:
			[YA] Asset tag: XYZ01234567
			[RW] Read-write area: 107 byte(s) free
		End
	Capabilities: [58] MSI: Enable+ Count=1/8 Maskable- 64bit+
		Address: 00000000fee0400c  Data: 4122
	Capabilities: [d0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [13c v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [160 v1] Device Serial Number 00-22-19-ff-fe-27-cd-f8
	Capabilities: [16c v1] Power Budgeting <?>
	Kernel driver in use: tg3
	Kernel modules: tg3

02:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 21)
	Subsystem: Dell Device 01eb
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 28
	NUMA node: 0
	Region 0: Memory at efff0000 (64-bit, non-prefetchable) [size=64K]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] Vital Product Data
		Product Name: Broadcom NetXtreme Gigabit Ethernet Controller
		Read-only fields:
			[PN] Part number: BCM95721
			[EC] Engineering changes: 106679-15
			[SN] Serial number: 0123456789
			[MN] Manufacture ID: 31 34 65 34
			[RV] Reserved: checksum good, 28 byte(s) reserved
		Read/write fields:
			[YA] Asset tag: XYZ01234567
			[RW] Read-write area: 107 byte(s) free
		End
	Capabilities: [58] MSI: Enable+ Count=1/8 Maskable- 64bit+
		Address: 00000000fee5500c  Data: 4142
	Capabilities: [d0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [13c v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [160 v1] Device Serial Number 00-22-19-ff-fe-27-cd-f9
	Capabilities: [16c v1] Power Budgeting <?>
	Kernel driver in use: tg3
	Kernel modules: tg3

03:0d.0 PCI bridge: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge (rev c0) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 256 bytes
	NUMA node: 0
	Bus: primary=03, secondary=04, subordinate=04, sec-latency=64
	I/O behind bridge: 0000f000-00000fff [empty]
	Memory behind bridge: fff00000-000fffff [empty]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [empty]
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [90] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=conv
		Status: Dev=03:0d.0 64bit+ 133MHz+ SCD- USC- SCO- SRD-
		Upstream: Capacity=8 CommitmentLimit=8
		Downstream: Capacity=8 CommitmentLimit=8
	Capabilities: [88] Subsystem: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge
	Kernel modules: shpchp

03:0e.0 IDE interface: Broadcom BCM5785 [HT1000] SATA (PATA/IDE Mode) (prog-if 8f [Master SecP SecO PriP PriO])
	Subsystem: Dell Device 01eb
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	Interrupt: pin A routed to IRQ 10
	NUMA node: 0
	Region 0: I/O ports at ecb0 [size=8]
	Region 1: I/O ports at eca0 [size=4]
	Region 2: I/O ports at ecb8 [size=8]
	Region 3: I/O ports at eca4 [size=4]
	Region 4: I/O ports at ece0 [size=16]
	Region 5: Memory at efdfe000 (32-bit, non-prefetchable) [size=8K]
	[virtual] Expansion ROM at efc00000 [disabled] [size=128K]
	Capabilities: [60] PCI-X non-bridge device
		Command: DPERE- ERO- RBC=512 OST=8
		Status: Dev=03:0e.0 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=512 DMOST=8 DMCRS=32 RSCEM- 266MHz- 533MHz-
	Capabilities: [90] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Kernel driver in use: sata_svw
	Kernel modules: sata_svw, pata_acpi, ata_generic
Comment 2 Wim ten Have 2017-07-05 07:16:45 EDT
From: Wim ten Have <wim.ten.have@oracle.com>
To: Sinan Kaya <okaya@codeaurora.org>
Cc: Bjorn Helgaas <bhelgaas@google.com>, "linux-pci@vger.kernel.org"  <linux-pci@vger.kernel.org>, Wim ten Have <wim.ten.have@oracle.com>
Subject: Re: Red Hat (Fedora) bug report 1467674 concerning your kernel  functional performance enhancements causing PCI Express crashes,
Date: Wed, 5 Jul 2017 13:13:12 +0200
X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.31; x86_64-redhat-linux-gnu)
Organization: Oracle Corporation

On Tue, 4 Jul 2017 21:00:03 -0400
Sinan Kaya <okaya@codeaurora.org> wrote:

> On 7/4/2017 6:25 PM, Sinan Kaya wrote:  
> > 
> > I can also see this in your lspci output. 
> > 
> > 00:08.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2) (prog-if 00 [Normal decode])
> > 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
> > 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > 	Latency: 0, Cache Line Size: 64 bytes
> > 	Interrupt: pin A routed to IRQ 19
> > 	NUMA node: 0
> > 	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
> > 	I/O behind bridge: 0000f000-00000fff [empty]
> > 	Memory behind bridge: efe00000-efefffff [size=1M]
> > 	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [empty]
> > 	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> > 	BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
> > 		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> > 	Capabilities: [a0] HyperTransport: MSI Mapping Enable+ Fixed-
> > 		Mapping Address Base: 00000000fee00000
> > 	Capabilities: [b0] Express (v1) Root Port (Slot-), MSI 00
> > 
> > I'll post a patch to apply extended tags to systems with PCI express v2 and higher
> > bridges only.  
> 
> Please give this patch a try. I can make the patch pretty and re-post if it works for you.   

    Howdy,

  I setup your patch under an /usr/src/kernel/rpmbuild tree for current
  "kernel-4.11.8-200.fc25.src.rpm" and made below change to kernel.spec
  file together with an rpmbuild -ba cycle on SPECS/kernel.spec.

  Your patch under the SOURCE/PATCH tree

	<wtenhave@hagen:55> ls -l SOURCES/0001-pci-do-not-enable-extended-tags-on-pre-dated-v1.x-sy.patch 
	-rw-r--r-- 1 wtenhave users 2658 Jul  5 09:45 SOURCES/0001-pci-do-not-enable-extended-tags-on-pre-dated-v1.x-sy.patch

  Change to the kernel.spec file

	<wtenhave@hagen:56> rcsdiff -u SPECS/kernel.spec
	===================================================================
	RCS file: SPECS/kernel.spec,v
	retrieving revision 1.1
	diff -u -r1.1 SPECS/kernel.spec
	--- SPECS/kernel.spec	2017/07/05 07:54:17	1.1
	+++ SPECS/kernel.spec	2017/07/05 07:54:20
	@@ -635,6 +635,9 @@
	 # rhbz 1459326
	 Patch683: RFC-audit-fix-a-race-condition-with-the-auditd-tracking-code.patch
	 
	+# rhbz 1467674
	+Patch700: 0001-pci-do-not-enable-extended-tags-on-pre-dated-v1.x-sy.patch
	+
	 # END OF PATCH DEFINITIONS
	 
	 %endif

  From an 'rpmbuild -ba kernel.spec' nicely proceeded and generated all
  package.  They all installed and the system boots and works like a charm!

> You should be seeing messages like this during boot.
> 
> [    3.949621] pci 0003:01:00.0: clearing extended tags capability
> [    3.959540] pci 0003:01:00.1: clearing extended tags capability
> [    3.969454] pci 0003:01:00.2: clearing extended tags capability
> [    3.979373] pci 0003:01:00.3: clearing extended tags capability
> [    3.989290] pci 0003:01:00.4: clearing extended tags capability  

  Correct ... see excerpt below.

	[    0.000000] Linux version 4.11.8-200.fc25.x86_64 (root@hagen) (gcc version 6.3.1 20161221 (Red Hat 6.3.1-1) (GCC) ) #1 SMP Wed Jul 5 10:37:18 CEST 2017
	[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.11.8-200.fc25.x86_64 root=/dev/mapper/fedora_hagen-root ro rd.lvm.lv=fedora_hagen/root rd.lvm.lv=fedora_hagen/swap audit=0
	[    0.000000] x86/fpu: x87 FPU will use FXSAVE
	[    0.000000] e820: BIOS-provided physical RAM map:
		...
	[    0.532911] PCI host bridge to bus 0000:00
	[    0.533104] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
	[    0.533301] pci_bus 0000:00: root bus resource [io  0xd000-0xefff window]
	[    0.533501] pci_bus 0000:00: root bus resource [io  0x0d00-0x0fff window]
	[    0.533699] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
	[    0.534039] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xf1ffffff window]
	[    0.534376] pci_bus 0000:00: root bus resource [mem 0xefb00000-0xefffffff window]
	[    0.534722] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe7ffffff window]
	[    0.535060] pci_bus 0000:00: root bus resource [bus 00-fd]
	[    0.535264] pci 0000:00:01.0: [1166:0036] type 01 class 0x060400
	[    0.535299] pci 0000:00:01.0: Enabling HT MSI Mapping
	[    0.535552] pci 0000:00:01.0: System wakeup disabled by ACPI
	[    0.535786] pci 0000:00:02.0: [1166:0205] type 00 class 0x060000
	[    0.535882] pci 0000:00:02.1: [1166:0214] type 00 class 0x01018a
	[    0.535904] pci 0000:00:02.1: reg 0x10: [io  0x01f0-0x01f7]
	[    0.535912] pci 0000:00:02.1: reg 0x14: [io  0x03f4-0x03f7]
	[    0.535920] pci 0000:00:02.1: reg 0x18: [io  0x0170-0x0177]
	[    0.535929] pci 0000:00:02.1: reg 0x1c: [io  0x0374-0x0377]
	[    0.535937] pci 0000:00:02.1: reg 0x20: [io  0x08c0-0x08cf]
	[    0.535956] pci 0000:00:02.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
	[    0.536154] pci 0000:00:02.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
	[    0.536350] pci 0000:00:02.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
	[    0.536551] pci 0000:00:02.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
	[    0.536814] pci 0000:00:02.2: [1166:0234] type 00 class 0x060100
	[    0.536951] pci 0000:00:03.0: [1166:0223] type 00 class 0x0c0310
	[    0.536967] pci 0000:00:03.0: reg 0x10: [mem 0xefbed000-0xefbedfff]
	[    0.536976] pci 0000:00:03.0: reg 0x14: [io  0xd000-0xd0ff]
	[    0.537089] pci 0000:00:03.1: [1166:0223] type 00 class 0x0c0310
	[    0.537104] pci 0000:00:03.1: reg 0x10: [mem 0xefbee000-0xefbeefff]
	[    0.537113] pci 0000:00:03.1: reg 0x14: [io  0xd400-0xd4ff]
	[    0.537222] pci 0000:00:03.2: [1166:0223] type 00 class 0x0c0320
	[    0.537237] pci 0000:00:03.2: reg 0x10: [mem 0xefbef000-0xefbeffff]
	[    0.537246] pci 0000:00:03.2: reg 0x14: [io  0xd800-0xd8ff]
	[    0.537311] pci 0000:00:03.2: supports D1 D2
	[    0.537312] pci 0000:00:03.2: PME# supported from D0 D1 D2 D3hot
	[    0.537373] pci 0000:00:04.0: [1002:515e] type 00 class 0x030000
	[    0.537389] pci 0000:00:04.0: reg 0x10: [mem 0xe0000000-0xe7ffffff pref]
	[    0.537398] pci 0000:00:04.0: reg 0x14: [io  0xdc00-0xdcff]
	[    0.537407] pci 0000:00:04.0: reg 0x18: [mem 0xefbf0000-0xefbfffff]
	[    0.537440] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
	[    0.537468] pci 0000:00:04.0: supports D1 D2
	[    0.537513] pci 0000:00:07.0: [1166:0140] type 01 class 0x060400
	[    0.537550] pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
	[    0.537603] pci 0000:00:08.0: [1166:0142] type 01 class 0x060400
	[    0.537639] pci 0000:00:08.0: PME# supported from D0 D3hot D3cold
	[    0.537666] pci 0000:00:08.0: System wakeup disabled by ACPI
	[    0.537893] pci 0000:00:09.0: [1166:0144] type 01 class 0x060400
	[    0.537936] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
	[    0.537964] pci 0000:00:09.0: System wakeup disabled by ACPI
	[    0.538187] pci 0000:00:0a.0: [1166:0142] type 01 class 0x060400
	[    0.538221] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
	[    0.538248] pci 0000:00:0a.0: System wakeup disabled by ACPI
	[    0.538478] pci 0000:00:0b.0: [1166:0144] type 01 class 0x060400
	[    0.538512] pci 0000:00:0b.0: PME# supported from D0 D3hot D3cold
	[    0.538577] pci 0000:00:18.0: [1022:1200] type 00 class 0x060000
	[    0.538629] pci 0000:00:18.1: [1022:1201] type 00 class 0x060000
	[    0.538683] pci 0000:00:18.2: [1022:1202] type 00 class 0x060000
	[    0.538729] pci 0000:00:18.3: [1022:1203] type 00 class 0x060000
	[    0.538778] pci 0000:00:18.4: [1022:1204] type 00 class 0x060000
	[    0.538827] pci 0000:00:19.0: [1022:1200] type 00 class 0x060000
	[    0.538885] pci 0000:00:19.1: [1022:1201] type 00 class 0x060000
	[    0.538945] pci 0000:00:19.2: [1022:1202] type 00 class 0x060000
	[    0.538993] pci 0000:00:19.3: [1022:1203] type 00 class 0x060000
	[    0.539045] pci 0000:00:19.4: [1022:1204] type 00 class 0x060000
	[    0.539152] pci 0000:03:0d.0: [1166:0104] type 01 class 0x060400
	[    0.539228] pci 0000:03:0e.0: [1166:024b] type 00 class 0x01018f
	[    0.539238] pci 0000:03:0e.0: reg 0x10: [io  0xecb0-0xecb7]
	[    0.539244] pci 0000:03:0e.0: reg 0x14: [io  0xeca0-0xeca3]
	[    0.539250] pci 0000:03:0e.0: reg 0x18: [io  0xecb8-0xecbf]
	[    0.539255] pci 0000:03:0e.0: reg 0x1c: [io  0xeca4-0xeca7]
	[    0.539261] pci 0000:03:0e.0: reg 0x20: [io  0xece0-0xecef]
	[    0.539267] pci 0000:03:0e.0: reg 0x24: [mem 0xefdfe000-0xefdfffff]
	[    0.539273] pci 0000:03:0e.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
	[    0.539337] pci 0000:00:01.0: PCI bridge to [bus 03-04]
	[    0.539537] pci 0000:00:01.0:   bridge window [io  0xe000-0xefff]
	[    0.539540] pci 0000:00:01.0:   bridge window [mem 0xefc00000-0xefdfffff]
	[    0.539598] pci 0000:03:0d.0: PCI bridge to [bus 04]
	[    0.539828] pci 0000:00:07.0: PCI bridge to [bus 05]
	[    0.540068] pci 0000:01:00.0: [14e4:1659] type 00 class 0x020000
	[    0.540087] pci 0000:01:00.0: reg 0x10: [mem 0xefef0000-0xefefffff 64bit]
	[    0.540185] pci 0000:01:00.0: PME# supported from D3hot D3cold
	[    0.542890] pci 0000:00:08.0: PCI bridge to [bus 01]
	[    0.543094] pci 0000:00:08.0:   bridge window [mem 0xefe00000-0xefefffff]
	[    0.543155] pci 0000:02:00.0: [14e4:1659] type 00 class 0x020000
	[    0.543175] pci 0000:02:00.0: reg 0x10: [mem 0xefff0000-0xefffffff 64bit]
	[    0.543277] pci 0000:02:00.0: PME# supported from D3hot D3cold
	[    0.545889] pci 0000:00:09.0: PCI bridge to [bus 02]
	[    0.546088] pci 0000:00:09.0:   bridge window [mem 0xeff00000-0xefffffff]
	[    0.546130] pci 0000:00:0a.0: PCI bridge to [bus 06]
	[    0.546351] pci 0000:00:0b.0: PCI bridge to [bus 07]
***>	[    0.546570] pci 0000:01:00.0: clearing extended tags capability
***>	[    0.546773] pci 0000:02:00.0: clearing extended tags capability  
		...

	<wtenhave@hagen:60> uname -a
	Linux hagen 4.11.8-200.fc25.x86_64 #1 SMP Wed Jul 5 10:37:18 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux

  I'll update RHT Bugzilla entry 1467674 with this info. 
  Hope someone picks this up for Fedora25 and Fedora26 (next week).

Regards,
- Wim.
Comment 3 Wim ten Have 2017-07-05 13:11:39 EDT
Below is email excerpt holding patch for final tested fix.

From: Sinan Kaya <okaya@codeaurora.org>
To: linux-pci@vger.kernel.org, timur@codeaurora.org, wim.ten.have@oracle.com
Cc: linux-arm-msm@vger.kernel.org, linux-arm-kernel@lists.infradead.org,         Sinan Kaya <okaya@codeaurora.org>, Bjorn Helgaas <bhelgaas@google.com>,         linux-kernel@vger.kernel.org
Subject: [PATCH] PCI: Do not enable extended tags on pre-dated (v1.x) systems
Date: Wed,  5 Jul 2017 09:19:08 -0400
X-Mailer: git-send-email 1.9.1

According to extended tags ECN document, all PCIe receivers are expected
to support extended tags support. It should be safe to enable extended
tags on endpoints without checking compatibility.

This assumption seems to be working fine except for the legacy systems.
The ECN has been written against PCIE spec version 2.0. Therefore, we need
to exclude all version 1.0 devices from this change as there is HW out
there that can't handle extended tags.

Note that the default value of Extended Tags Enable bit is implementation
specific. Therefore, we are clearing the bit by default when incompatible
HW is found without assuming that value is zero.

Reported-by: Wim ten Have <wim.ten.have@oracle.com>
Link: https://pcisig.com/sites/default/files/specification_documents/ECN_Extended_Tag_Enable_Default_05Sept2008_final.pdf
Link: https://bugzilla.redhat.com/show_bug.cgi?id=1467674
Fixes: 60db3a4d8cc9 ("PCI: Enable PCIe Extended Tags if supported")
Tested-by: Wim ten Have <wim.ten.have@oracle.com>
Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/probe.c | 52 +++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 45 insertions(+), 7 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 19c8950..5e39013 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1684,21 +1684,58 @@ static void program_hpp_type2(struct pci_dev *dev, struct hpp_type2 *hpp)
 	 */
 }
 
-static void pci_configure_extended_tags(struct pci_dev *dev)
+static bool pcie_bus_exttags_supported(struct pci_bus *bus)
+{
+	bool exttags_supported = true;
+	struct pci_dev *bridge;
+	int rc;
+	u16 flags;
+
+	bridge = bus->self;
+	while (bridge) {
+		if (pci_is_pcie(bridge)) {
+			rc = pcie_capability_read_word(bridge, PCI_EXP_FLAGS,
+						       &flags);
+			if (!rc && ((flags & PCI_EXP_FLAGS_VERS) < 2)) {
+				exttags_supported = false;
+				break;
+			}
+		}
+		if (!bridge->bus->parent)
+			break;
+		bridge = bridge->bus->parent->self;
+	}
+
+	return exttags_supported;
+}
+
+static int pcie_bus_configure_exttags(struct pci_dev *dev, void *data)
 {
 	u32 dev_cap;
 	int ret;
+	bool supported;
 
 	if (!pci_is_pcie(dev))
-		return;
+		return 0;
 
 	ret = pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &dev_cap);
 	if (ret)
-		return;
+		return 0;
 
-	if (dev_cap & PCI_EXP_DEVCAP_EXT_TAG)
-		pcie_capability_set_word(dev, PCI_EXP_DEVCTL,
-					 PCI_EXP_DEVCTL_EXT_TAG);
+	if (dev_cap & PCI_EXP_DEVCAP_EXT_TAG) {
+		supported = pcie_bus_exttags_supported(dev->bus);
+
+		if (supported) {
+			dev_dbg(&dev->dev, "setting extended tags capability\n");
+			pcie_capability_set_word(dev, PCI_EXP_DEVCTL,
+						 PCI_EXP_DEVCTL_EXT_TAG);
+		} else {
+			dev_dbg(&dev->dev, "clearing extended tags capability\n");
+			pcie_capability_clear_word(dev, PCI_EXP_DEVCTL,
+						   PCI_EXP_DEVCTL_EXT_TAG);
+		}
+	}
+	return 0;
 }
 
 static void pci_configure_device(struct pci_dev *dev)
@@ -1707,7 +1744,6 @@ static void pci_configure_device(struct pci_dev *dev)
 	int ret;
 
 	pci_configure_mps(dev);
-	pci_configure_extended_tags(dev);
 
 	memset(&hpp, 0, sizeof(hpp));
 	ret = pci_get_hp_params(dev, &hpp);
@@ -2224,6 +2260,8 @@ void pcie_bus_configure_settings(struct pci_bus *bus)
 
 	pcie_bus_configure_set(bus->self, &smpss);
 	pci_walk_bus(bus, pcie_bus_configure_set, &smpss);
+
+	pci_walk_bus(bus, pcie_bus_configure_exttags, NULL);
 }
 EXPORT_SYMBOL_GPL(pcie_bus_configure_settings);
 
-- 
1.9.1
Comment 4 Sinan Kaya 2017-07-11 10:04:37 EDT
Can you also attach lspci -nn output from your system? PCI maintainer asked for a quirk to be implemented for your machine. Quirks are identified per vendor-id/device-id pairs.
Comment 5 Wim ten Have 2017-07-11 10:25:16 EDT
<wtenhave@hagen:53> sudo lspci -nn
00:01.0 PCI bridge [0604]: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge [1166:0036]
00:02.0 Host bridge [0600]: Broadcom BCM5785 [HT1000] Legacy South Bridge [1166:0205]
00:02.1 IDE interface [0101]: Broadcom BCM5785 [HT1000] IDE [1166:0214]
00:02.2 ISA bridge [0601]: Broadcom BCM5785 [HT1000] LPC [1166:0234]
00:03.0 USB controller [0c03]: Broadcom BCM5785 [HT1000] USB [1166:0223] (rev 01)
00:03.1 USB controller [0c03]: Broadcom BCM5785 [HT1000] USB [1166:0223] (rev 01)
00:03.2 USB controller [0c03]: Broadcom BCM5785 [HT1000] USB [1166:0223] (rev 01)
00:04.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 [1002:515e] (rev 02)
00:07.0 PCI bridge [0604]: Broadcom HT2100 PCI-Express Bridge [1166:0140] (rev a2)
00:08.0 PCI bridge [0604]: Broadcom HT2100 PCI-Express Bridge [1166:0142] (rev a2)
00:09.0 PCI bridge [0604]: Broadcom HT2100 PCI-Express Bridge [1166:0144] (rev a2)
00:0a.0 PCI bridge [0604]: Broadcom HT2100 PCI-Express Bridge [1166:0142] (rev a2)
00:0b.0 PCI bridge [0604]: Broadcom HT2100 PCI-Express Bridge [1166:0144] (rev a2)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration [1022:1200]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map [1022:1201]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller [1022:1202]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control [1022:1203]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control [1022:1204]
00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration [1022:1200]
00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map [1022:1201]
00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller [1022:1202]
00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control [1022:1203]
00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control [1022:1204]
01:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5721 Gigabit Ethernet PCI Express [14e4:1659] (rev 21)
02:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5721 Gigabit Ethernet PCI Express [14e4:1659] (rev 21)
03:0d.0 PCI bridge [0604]: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge [1166:0104] (rev c0)
03:0e.0 IDE interface [0101]: Broadcom BCM5785 [HT1000] SATA (PATA/IDE Mode) [1166:024b]
Comment 6 Fedora End Of Life 2017-11-16 14:28:22 EST
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Comment 7 Fedora End Of Life 2017-12-12 05:26:36 EST
Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.