RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1295423 - Unstable network link using bond mode = 4
Summary: Unstable network link using bond mode = 4
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: 7.3
Assignee: Jarod Wilson
QA Contact: Amit Supugade
URL:
Whiteboard:
Depends On:
Blocks: 1274397
TreeView+ depends on / blocked
 
Reported: 2016-01-04 12:55 UTC by HWSD
Modified: 2016-03-22 20:47 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-02-15 20:46:36 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description HWSD 2016-01-04 12:55:07 UTC 
Description of problem:
network link is unstable using a bond in mode 4.

Version-Release number of selected component (if applicable):

kernel-3.10.0-327.3.1.el7.x86_64 -> bond mode 4 doesn't work (if I detach one network cable the network is stable)
kernel-3.10.0-229.20.1.el7.x86_64 -> bond mode 4 works fine 

My network setup:

Switch D-Link DGS-1100-08 REV. A1 Firmware 1.10.033 

- Nics on bond specs:
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
        Subsystem: ASUSTeK Computer Inc. Motherboard
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at df200000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at e000 [size=32]
        Memory at df220000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: e1000e 

----------------

[root@ovirt01 ~]# ifconfig
DMZ: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        txqueuelen 0  (Ethernet)
        RX packets 43546  bytes 2758816 (2.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

LAN_HAW: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        txqueuelen 0  (Ethernet)
        RX packets 2090262  bytes 201078292 (191.7 MiB)
        RX errors 0  dropped 86  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 1500
        txqueuelen 0  (Ethernet)
        RX packets 2408059  bytes 456371629 (435.2 MiB)
        RX errors 0  dropped 185  overruns 0  frame 0
        TX packets 118966  bytes 14862549 (14.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bond0.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        txqueuelen 0  (Ethernet)
        RX packets 2160985  bytes 210157656 (200.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bond0.3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        txqueuelen 0  (Ethernet)
        RX packets 151195  bytes 185253584 (176.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 118663  bytes 13857950 (13.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        txqueuelen 1000  (Ethernet)
        RX packets 708141  bytes 95034564 (90.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16714  bytes 5193108 (4.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xdf200000-df220000

enp5s0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        txqueuelen 1000  (Ethernet)
        RX packets 1699934  bytes 361339105 (344.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 102252  bytes 9669441 (9.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xdf100000-df120000

enp6s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        txqueuelen 1000  (Ethernet)
        RX packets 2525232  bytes 362345893 (345.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 388452  bytes 208145492 (198.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 0  (Local Loopback)
        RX packets 116465661  bytes 1515059255942 (1.3 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 116465661  bytes 1515059255942 (1.3 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ovirtmgmt: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.50  netmask 255.255.255.0  broadcast 192.168.1.255
        txqueuelen 0  (Ethernet)
        RX packets 3784298  bytes 555536509 (529.8 MiB)
        RX errors 0  dropped 86  overruns 0  frame 0
        TX packets 1737669  bytes 1401650369 (1.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        txqueuelen 500  (Ethernet)
        RX packets 558574  bytes 107521742 (102.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1316892  bytes 487764500 (465.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        txqueuelen 500  (Ethernet)
        RX packets 42282  bytes 7373007 (7.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40498  bytes 17598215 (16.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        txqueuelen 500  (Ethernet)
        RX packets 79388  bytes 16807917 (16.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 164596  bytes 183858757 (175.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 

-------------------------

[root@ovirt01 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 1
        Actor Key: 9
        Partner Key: 1
        Partner Mac Address: 00:00:00:00:00:00

Slave Interface: enp4s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: 
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 1
Partner Churned Count: 2
details actor lacp pdu:
    system priority: 65535
    port key: 9
    port priority: 255
    port number: 1
    port state: 69
details partner lacp pdu:
    system priority: 65535
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

Slave Interface: enp5s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    port key: 9
    port priority: 255
    port number: 2
    port state: 77
details partner lacp pdu:
    system priority: 65535
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1
Comment 2 HWSD 2016-01-04 13:29:27 UTC 
Bond info when using kernel-3.10.0-229.20.1.el7.x86_64

[root@ovirt01 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 1
        Actor Key: 17
        Partner Key: 1
        Partner Mac Address: 00:00:00:00:00:00

Slave Interface: enp4s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: enp5s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 
Aggregator ID: 2
Slave queue ID: 0
Comment 3 Jarod Wilson 2016-01-12 19:29:40 UTC 
Do we have any idea if this behaves the same with a current upstream kernel? Would like to get an idea if this is something that needs to be fixed upstream, or if I'm just hunting for an already-upstream fix. I've got a bonding driver refresh in the queue for 7.3 that could potentially already carry the fix as well.

Also, in what way is the connection unstable? I see dropped packets on bond0, but is there any additional relevant log spew, or insight that might help narrow in on the problem?
Comment 4 Jarod Wilson 2016-01-12 19:32:50 UTC 
Offhand, this looks like the patch from upstream most likely to have an impact here:

    commit f6a69a8fd6c009c0f5958f769dd4165773c75695
    Author: Mahesh Bandewar <maheshb>
    Date:   Mon Mar 30 14:31:16 2015 -0700

        bonding: Fix another case of LACPDU not sent on slave

        When mii-mon discovers that the link is up, it will call
        bond_3ad_handle_link_change() but we forget to add the LACP_ENABLED
        flag when we discover the speed and duplex for the slave link are
        normal.

        Change-Id: Ie8b268ecfeea0f99bf9fdcd72706c0653f9d9e49
        Signed-off-by: Mahesh Bandewar <maheshb>
        Signed-off-by: Andy Gospodarek <gospo>
        Signed-off-by: David S. Miller <davem>
Comment 5 HWSD 2016-01-12 21:53:29 UTC 
(In reply to Jarod Wilson from comment #3)
> Do we have any idea if this behaves the same with a current upstream kernel?
> Would like to get an idea if this is something that needs to be fixed
> upstream, or if I'm just hunting for an already-upstream fix. I've got a
> bonding driver refresh in the queue for 7.3 that could potentially already
> carry the fix as well.
> 
> Also, in what way is the connection unstable? I see dropped packets on
> bond0, but is there any additional relevant log spew, or insight that might
> help narrow in on the problem?

I've no idea about there are differences with upstream kernel. I used precompiled packages.

I use the bond to serve networking of kvm virtual machines.

I made some tests using kernel  3.10.0-327.4.4.el7.x86_64.
I did a TCP dump on the bond and pinged VMs from the network.

When I have two netwok cables connected I can se ARP requests from the network but not ARP replyes from VMs.
When I detach one network cable I can see ARP requests and ARP replyes (and networking on VM works)

When there are all two netwoks cabels connected VMs are isolated from the network. After some minutes network work fine for a cuple of minutes, after VMs are another time isolated for minutes (5/10).

I can try more detailed debug if you need.
Comment 6 Jarod Wilson 2016-01-12 22:02:12 UTC 
I've got a setup here in a VM that is able to talk to the network okay, but I do see the dropped packet counter for bond0 increasing, and in dmesg, I'm seeing:

bond0: An illegal loopback occurred on adapter (enp0s9)
Check the configuration to verify that all adapters are connected to 802.3ad compliant switch ports

This repeats quite often, for both devices in the bond. Are you seeing anything similar, or is this possibly just an issue with my VM setup's virtual switch not knowing how to be 802.3ad-compliant?
Comment 7 HWSD 2016-01-12 22:10:29 UTC 
(In reply to Jarod Wilson from comment #6)
> I've got a setup here in a VM that is able to talk to the network okay, but
> I do see the dropped packet counter for bond0 increasing, and in dmesg, I'm
> seeing:
> 
> bond0: An illegal loopback occurred on adapter (enp0s9)
> Check the configuration to verify that all adapters are connected to 802.3ad
> compliant switch ports
> 
> This repeats quite often, for both devices in the bond. Are you seeing
> anything similar, or is this possibly just an issue with my VM setup's
> virtual switch not knowing how to be 802.3ad-compliant?

I don't see any warning in dmesg or dropped packet counter increasing.
I think that you see could be related to your virtual switch.
Comment 8 Dan Kenigsberg 2016-01-18 13:19:40 UTC 
Jarod, can you suggest any debugging that the reported can enable?
Comment 9 Jarod Wilson 2016-01-23 02:16:29 UTC 
To date, I'm still unable to reproduce this. I've set up a bond on sfc and bnx2x NICs, and can't duplicate the problem. The link is rock-solid, with outstanding throughput:

# ./netperf -l 1000000 -j -H shost
MIGRATED TCP STREAM TEST from ::0 (::) port 0 AF_INET6 to shost () port 0 AF_INET6
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    14327.34    916.31


# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
	Aggregator ID: 1
	Number of ports: 1
	Actor Key: 13
	Partner Key: 1
	Partner Mac Address: 00:00:00:00:00:00

Slave Interface: p5p1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0f:53:16:3e:b0
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    port key: 13
    port priority: 255
    port number: 1
    port state: 77
details partner lacp pdu:
    system priority: 65535
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

Slave Interface: p5p2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0f:53:16:3e:b1
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    port key: 13
    port priority: 255
    port number: 2
    port state: 69
details partner lacp pdu:
    system priority: 65535
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

Slave Interface: p7p1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0a:f7:30:9f:50
Slave queue ID: 0
Aggregator ID: 3
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    port key: 13
    port priority: 255
    port number: 3
    port state: 69
details partner lacp pdu:
    system priority: 65535
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1
Comment 10 Jarod Wilson 2016-01-23 02:18:28 UTC 
Probably need to enable dynamic debugging in the network stack and/or the NIC driver to get some better idea of where things are falling down. This looks like it might be specific to either the NIC driver or the specific NICs though, given that I can't reproduce this with multiple other NICs on my end.
Comment 12 Dan Kenigsberg 2016-02-01 13:13:51 UTC 
(In reply to Jarod Wilson from comment #10)
> Probably need to enable dynamic debugging in the network stack and/or the
> NIC driver to get some better idea of where things are falling down. This
> looks like it might be specific to either the NIC driver or the specific
> NICs though, given that I can't reproduce this with multiple other NICs on
> my end.

Jarod, would you please give laymen-level details on who to enabled the dynamic debugging that can be helpful to you? Currently we have a single report about this as a el7.2 regression, but others may be lurking.

s.danzi, would you specify your lspci -vvv in case this is related to your specific hardware?
Comment 13 HWSD 2016-02-01 14:51:19 UTC 
I don't know if it's closed to my hardware. This is my only test machine.

[root@ovirt01 ~]# lspci -vvv
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
        Subsystem: ASUSTeK Computer Inc. Device 8497
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
        Latency: 0
        Capabilities: [e0] Vendor Specific Information: Len=0c <?>
        Kernel driver in use: ie31200_edac

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: fff00000-000fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [88] Subsystem: ASUSTeK Computer Inc. Device 8497
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee002d8  Data: 0000
        Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
                        ClockPM- Surprise- LLActRep- BwNot+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #1, PowerLimit 75.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending+ InProgress-
        Capabilities: [140 v1] Root Complex Link
                Desc:   PortNumber=02 ComponentID=01 EltType=Config
                Link0:  Desc:   TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
                        Addr:   00000000fed19000
        Capabilities: [d94 v1] #19
        Kernel driver in use: pcieport

00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: fff00000-000fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [88] Subsystem: ASUSTeK Computer Inc. Device 8497
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee00318  Data: 0000
        Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #5, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
                        ClockPM- Surprise- LLActRep- BwNot+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #4, PowerLimit 75.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending+ InProgress-
        Capabilities: [140 v1] Root Complex Link
                Desc:   PortNumber=05 ComponentID=01 EltType=Config
                Link0:  Desc:   TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
                        Addr:   00000000fed19000
        Kernel driver in use: pcieport

00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
        Subsystem: ASUSTeK Computer Inc. Device 8497
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at df304000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
                AFCap: TP+ FLR+
                AFCtrl: FLR-
                AFStatus: TP-
        Kernel driver in use: ehci-pci

00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: fff00000-000fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #1, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #0, PowerLimit 25.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8497
        Capabilities: [a0] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: pcieport

00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: df200000-df2fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #4, PowerLimit 10.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8497
        Capabilities: [a0] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: pcieport

00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: df100000-df1fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #5, PowerLimit 10.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8497
        Capabilities: [a0] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: pcieport

00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
        Subsystem: ASUSTeK Computer Inc. Device 8497
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 23
        Region 0: Memory at df303000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
                AFCap: TP+ FLR+
                AFCtrl: FLR-
                AFStatus: TP-
        Kernel driver in use: ehci-pci

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Bus: primary=00, secondary=06, subordinate=06, sec-latency=32
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: de000000-df0fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] Subsystem: ASUSTeK Computer Inc. Device 8497

00:1f.0 ISA bridge: Intel Corporation C204 Chipset Family LPC Controller (rev 05)
        Subsystem: ASUSTeK Computer Inc. Device 8497
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Capabilities: [e0] Vendor Specific Information: Len=0c <?>
        Kernel driver in use: lpc_ich

00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05) (prog-if 01 [AHCI 1.0])
        Subsystem: ASUSTeK Computer Inc. Device 8498
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin B routed to IRQ 30
        Region 0: I/O ports at f070 [size=8]
        Region 1: I/O ports at f060 [size=4]
        Region 2: I/O ports at f050 [size=8]
        Region 3: I/O ports at f040 [size=4]
        Region 4: I/O ports at f020 [size=32]
        Region 5: Memory at df302000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee00418  Data: 0000
        Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
        Capabilities: [b0] PCI Advanced Features
                AFCap: TP+ FLR+
                AFCtrl: FLR-
                AFStatus: TP-
        Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
        Subsystem: ASUSTeK Computer Inc. Device 8497
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin C routed to IRQ 18
        Region 0: Memory at df301000 (64-bit, non-prefetchable) [size=256]
        Region 4: I/O ports at f000 [size=32]

04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
        Subsystem: ASUSTeK Computer Inc. Motherboard
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at df200000 (32-bit, non-prefetchable) [size=128K]
        Region 2: I/O ports at e000 [size=32]
        Region 3: Memory at df220000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [e0] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <128ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [140 v1] Device Serial Number 40-16-7e-ff-ff-36-84-f1
        Kernel driver in use: e1000e

05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
        Subsystem: ASUSTeK Computer Inc. Motherboard
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 17
        Region 0: Memory at df100000 (32-bit, non-prefetchable) [size=128K]
        Region 2: I/O ports at d000 [size=32]
        Region 3: Memory at df120000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [e0] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <128ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [140 v1] Device Serial Number 40-16-7e-ff-ff-36-84-f2
        Kernel driver in use: e1000e

06:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8169/8110 Family PCI Gigabit Ethernet NIC
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64 (8000ns min, 16000ns max), Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 20
        Region 0: I/O ports at c000 [size=256]
        Region 1: Memory at df040000 (32-bit, non-prefetchable) [size=256]
        Expansion ROM at df020000 [disabled] [size=128K]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: r8169

06:05.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device 8373
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 23
        Region 0: Memory at de000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at df000000 (32-bit, non-prefetchable) [size=128K]
        Region 2: I/O ports at c100 [size=128]
        Expansion ROM at <unassigned> [disabled]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: ast
Comment 14 Jarod Wilson 2016-02-02 00:04:25 UTC 
(In reply to Dan Kenigsberg from comment #12)
> (In reply to Jarod Wilson from comment #10)
> > Probably need to enable dynamic debugging in the network stack and/or the
> > NIC driver to get some better idea of where things are falling down. This
> > looks like it might be specific to either the NIC driver or the specific
> > NICs though, given that I can't reproduce this with multiple other NICs on
> > my end.
> 
> Jarod, would you please give laymen-level details on who to enabled the
> dynamic debugging that can be helpful to you? Currently we have a single
> report about this as a el7.2 regression, but others may be lurking.

It's pretty well covered here:

https://www.kernel.org/doc/Documentation/dynamic-debug-howto.txt

I'd probably go with enabling it for at least the bonding and e1000e driver to start out, in this particular case, which can be done by simply adding some bits to /etc/modprobe.d/debug.conf (and then rebooting):

options bonding dyndbg
options e1000e dyndbg

We may need some additional bits enabled for the likes of net/core/dev.c, but let's start with those...
Comment 15 HWSD 2016-02-03 17:29:22 UTC 
Let me know if you need any specific debug.

[root@ovirt01 ~]# cat /sys/kernel/debug/dynamic_debug/control | grep bond
drivers/net/bonding/bond_main.c:4375 [bonding]bond_check_params =_ "Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details\012"
drivers/net/bonding/bond_main.c:3008 [bonding]bond_netdev_event =_ "IFF_SLAVE\012"
drivers/net/bonding/bond_main.c:3003 [bonding]bond_netdev_event =_ "IFF_MASTER\012"
drivers/net/bonding/bond_main.c:2997 [bonding]bond_netdev_event =_ "event: %lx\012"
drivers/net/bonding/bond_main.c:4492 [bonding]bond_init =_ "Begin bond_init\012"
drivers/net/bonding/bond_main.c:2232 [bonding]bond_arp_send =_ "outer reg: proto %X vid %X\012"
drivers/net/bonding/bond_main.c:2222 [bonding]bond_arp_send =_ "inner tag: proto %X vid %X\012"
drivers/net/bonding/bond_main.c:2210 [bonding]bond_arp_send =_ "arp %d on slave %s: dst %pI4 src %pI4\012"
drivers/net/bonding/bond_main.c:2323 [bonding]bond_arp_send_all =_ "no path to arp_ip_target %pI4 via rt.dev %s\012"
drivers/net/bonding/bond_main.c:2249 [bonding]bond_arp_send_all =_ "basa: target %pI4\012"
drivers/net/bonding/bond_main.c:3551 [bonding]bond_set_mac_address =_ "unwind err %d dev %s\012"
drivers/net/bonding/bond_main.c:3528 [bonding]bond_set_mac_address =_ "err %d %s\012"
drivers/net/bonding/bond_main.c:3519 [bonding]bond_set_mac_address =_ "slave %p %s\012"
drivers/net/bonding/bond_main.c:3506 [bonding]bond_set_mac_address =_ "bond=%p\012"
drivers/net/bonding/bond_main.c:3321 [bonding]bond_do_ioctl =_ "slave_dev->name=%s:\012"
drivers/net/bonding/bond_main.c:3316 [bonding]bond_do_ioctl =_ "slave_dev=%p:\012"
drivers/net/bonding/bond_main.c:3254 [bonding]bond_do_ioctl =_ "bond_ioctl: cmd=%d\012"
drivers/net/bonding/bond_main.c:3481 [bonding]bond_change_mtu =_ "unwind err %d dev %s\012"
drivers/net/bonding/bond_main.c:3461 [bonding]bond_change_mtu =_ "err %d %s\012"
drivers/net/bonding/bond_main.c:3447 [bonding]bond_change_mtu =_ "s %p c_m %p\012"
drivers/net/bonding/bond_main.c:3443 [bonding]bond_change_mtu =_ "bond=%p, new_mtu=%d\012"
drivers/net/bonding/bond_main.c:3744 [bonding]bond_update_slave_arr =_ "bond_3ad_get_active_agg_info failed\012"
drivers/net/bonding/bond_main.c:2349 [bonding]bond_validate_arp =_ "bva: sip %pI4 not found in targets\012"
drivers/net/bonding/bond_main.c:2342 [bonding]bond_validate_arp =_ "bva: sip %pI4 tip %pI4 not found\012"
drivers/net/bonding/bond_main.c:2404 [bonding]bond_arp_rcv =_ "bond_arp_rcv: %s/%d av %d sv %d sip %pI4 tip %pI4\012"
drivers/net/bonding/bond_main.c:2377 [bonding]bond_arp_rcv =_ "bond_arp_rcv: skb->dev %s\012"
drivers/net/bonding/bond_main.c:1737 [bonding]__bond_release_one =_ "cannot release %s\012"
drivers/net/bonding/bond_main.c:1628 [bonding]bond_enslave =_ "Error %d calling bond_sysfs_slave_add\012"
drivers/net/bonding/bond_main.c:1621 [bonding]bond_enslave =_ "Error %d calling bond_master_upper_dev_link\012"
drivers/net/bonding/bond_main.c:1615 [bonding]bond_enslave =_ "Error %d calling netdev_rx_handler_register\012"
drivers/net/bonding/bond_main.c:1581 [bonding]bond_enslave =_ "This slave is always active in trunk mode\012"
drivers/net/bonding/bond_main.c:1540 [bonding]bond_enslave =_ "Initial state of slave_dev is BOND_LINK_%s\012"
drivers/net/bonding/bond_main.c:1424 [bonding]bond_enslave =_ "Opening slave %s failed\012"
drivers/net/bonding/bond_main.c:1416 [bonding]bond_enslave =_ "Error %d calling set_mac_address\012"
drivers/net/bonding/bond_main.c:1397 [bonding]bond_enslave =_ "Error %d calling dev_set_mtu\012"
drivers/net/bonding/bond_main.c:1324 [bonding]bond_enslave =_ "change device type from %d to %d\012"
drivers/net/bonding/bond_main.c:1299 [bonding]bond_enslave =_ "%s is !NETIF_F_VLAN_CHALLENGED\012"
drivers/net/bonding/bond_main.c:1287 [bonding]bond_enslave =_ "%s is NETIF_F_VLAN_CHALLENGED\012"
drivers/net/bonding/bond_main.c:1274 [bonding]bond_enslave =_ "Error: Device was already enslaved\012"
drivers/net/bonding/bond_main.c:746 [bonding]bond_should_notify_peers =_ "bond_should_notify_peers: slave %s\012"
drivers/net/bonding/bond_main.c:619 [bonding]bond_set_dev_addr =_ "bond_dev=%p slave_dev=%p slave_dev->addr_len=%d\012"
drivers/net/bonding/bond_3ad.c:1862 [bonding]ad_marker_info_received =_ "Sent Marker Response on port %d\012"
drivers/net/bonding/bond_3ad.c:2306 [bonding]bond_3ad_rx_indication =_ "Received an unknown Marker subtype on slot %d\012"
drivers/net/bonding/bond_3ad.c:2300 [bonding]bond_3ad_rx_indication =_ "Received Marker Response on port %d\012"
drivers/net/bonding/bond_3ad.c:2294 [bonding]bond_3ad_rx_indication =_ "Received Marker Information on port %d\012"
drivers/net/bonding/bond_3ad.c:2278 [bonding]bond_3ad_rx_indication =_ "Received LACPDU on port %d slave %s\012"
drivers/net/bonding/bond_3ad.c:2427 [bonding]bond_3ad_handle_link_change =_ "Port %d changed link status to %s\012"
drivers/net/bonding/bond_3ad.c:2371 [bonding]bond_3ad_adapter_duplex_changed =_ "Port %d slave %s changed duplex\012"
drivers/net/bonding/bond_3ad.c:2337 [bonding]bond_3ad_adapter_speed_changed =_ "Port %d changed speed\012"
drivers/net/bonding/bond_3ad.c:1195 [bonding]ad_tx_machine =_ "Sent LACPDU on port %d\012"
drivers/net/bonding/bond_3ad.c:1817 [bonding]ad_enable_collecting_distributing =_ "Enabling port %d(LAG %d)\012"
drivers/net/bonding/bond_3ad.c:1837 [bonding]ad_disable_collecting_distributing =_ "Disabling port %d(LAG %d)\012"
drivers/net/bonding/bond_3ad.c:956 [bonding]ad_mux_machine =_ "Mux Machine: Port=%d (%s), Last State=%d, Curr State=%d\012"
drivers/net/bonding/bond_3ad.c:1448 [bonding]ad_port_selection_logic =_ "Port %d joined LAG %d(new LAG)\012"
drivers/net/bonding/bond_3ad.c:1403 [bonding]ad_port_selection_logic =_ "Port %d joined LAG %d(existing LAG)\012"
drivers/net/bonding/bond_3ad.c:1354 [bonding]ad_port_selection_logic =_ "Port %d left LAG %d\012"
drivers/net/bonding/bond_3ad.c:1276 [bonding]ad_periodic_machine =_ "Periodic Machine: Port=%d, Last State=%d, Curr State=%d\012"
drivers/net/bonding/bond_3ad.c:477 [bonding]__record_pdu =_ "%s partner sync=0\012"
drivers/net/bonding/bond_3ad.c:474 [bonding]__record_pdu =_ "%s partner sync=1\012"
drivers/net/bonding/bond_3ad.c:1074 [bonding]ad_rx_machine =_ "Rx Machine: Port=%d (%s), Last State=%d, Curr State=%d\012"
drivers/net/bonding/bond_3ad.c:1659 [bonding]ad_agg_selection_logic =_ "Agg=%d; P=%d; a k=%d; p k=%d; Ind=%d; Act=%d\012"
drivers/net/bonding/bond_3ad.c:1654 [bonding]ad_agg_selection_logic =_ "LAG %d chosen as the active LAG\012"
drivers/net/bonding/bond_3ad.c:1642 [bonding]ad_agg_selection_logic =_ "Agg=%d; P=%d; a k=%d; p k=%d; Ind=%d; Act=%d\012"
drivers/net/bonding/bond_3ad.c:1633 [bonding]ad_agg_selection_logic =_ "best ports %p slave %p %s\012"
drivers/net/bonding/bond_3ad.c:1630 [bonding]ad_agg_selection_logic =_ "best Agg=%d; P=%d; a k=%d; p k=%d; Ind=%d; Act=%d\012"
drivers/net/bonding/bond_3ad.c:734 [bonding]__update_lacpdu_from_port =_ "update lacpdu: %s, actor port state %x\012"
drivers/net/bonding/bond_3ad.c:2111 [bonding]bond_3ad_unbind_slave =_ "Unbinding port %d\012"
drivers/net/bonding/bond_3ad.c:2057 [bonding]bond_3ad_unbind_slave =_ "Some port(s) related to LAG %d - replacing with LAG %d\012"
drivers/net/bonding/bond_3ad.c:2020 [bonding]bond_3ad_unbind_slave =_ "Unbinding Link Aggregation Group %d\012"
drivers/net/bonding/bond_3ad.c:1712 [bonding]ad_clear_agg =_ "LAG %d was cleared\012"
drivers/net/bonding/bond_3ad.c:315 [bonding]__get_link_speed =_ "Port %d Received link speed %d update from adapter\012"
drivers/net/bonding/bond_3ad.c:348 [bonding]__get_duplex =_ "Port %d Received status NOT full duplex update from adapter\012"
drivers/net/bonding/bond_3ad.c:342 [bonding]__get_duplex =_ "Port %d Received status full duplex update from adapter\012"
drivers/net/bonding/bond_alb.c:713 [bonding]rlb_arp_xmit =_ "Server sent ARP Request packet\012"
drivers/net/bonding/bond_alb.c:693 [bonding]rlb_arp_xmit =_ "Server sent ARP Reply packet\012"
drivers/net/bonding/bond_alb.c:326 [bonding]rlb_arp_recv =_ "Server received an ARP Reply from client\012"

---------------

[root@ovirt01 ~]# cat /sys/kernel/debug/dynamic_debug/control | grep 1000
security/capability.c:1000 [capability]security_fixup_ops =_ "Had to override the file_send_sigiotask security operation with the default.\012"
drivers/net/ethernet/intel/e1000e/82571.c:129 [e1000e]e1000_init_phy_params_82571 =_ "PHY ID unknown: type = 0x%08x\012"
drivers/net/ethernet/intel/e1000e/82571.c:103 [e1000e]e1000_init_phy_params_82571 =_ "Error getting PHY ID\012"
drivers/net/ethernet/intel/e1000e/82571.c:1651 [e1000e]e1000_check_for_serdes_link_82571 =_ "ANYSTATE  -> AN_PROG\012"
drivers/net/ethernet/intel/e1000e/82571.c:1639 [e1000e]e1000_check_for_serdes_link_82571 =_ "ANYSTATE  -> DOWN\012"
drivers/net/ethernet/intel/e1000e/82571.c:1622 [e1000e]e1000_check_for_serdes_link_82571 =_ "ANYSTATE  -> DOWN\012"
drivers/net/ethernet/intel/e1000e/82571.c:1615 [e1000e]e1000_check_for_serdes_link_82571 =_ "DOWN      -> AN_PROG\012"
drivers/net/ethernet/intel/e1000e/82571.c:1600 [e1000e]e1000_check_for_serdes_link_82571 =_ "AN_PROG   -> FORCED_UP\012"
drivers/net/ethernet/intel/e1000e/82571.c:1594 [e1000e]e1000_check_for_serdes_link_82571 =_ "Error config flow control\012"
drivers/net/ethernet/intel/e1000e/82571.c:1580 [e1000e]e1000_check_for_serdes_link_82571 =_ "AN_PROG   -> DOWN\012"
drivers/net/ethernet/intel/e1000e/82571.c:1574 [e1000e]e1000_check_for_serdes_link_82571 =_ "AN_PROG   -> AN_UP\012"
drivers/net/ethernet/intel/e1000e/82571.c:1559 [e1000e]e1000_check_for_serdes_link_82571 =_ "FORCED_UP -> AN_PROG\012"
drivers/net/ethernet/intel/e1000e/82571.c:1540 [e1000e]e1000_check_for_serdes_link_82571 =_ "AN_UP     -> AN_PROG\012"
drivers/net/ethernet/intel/e1000e/82571.c:320 [e1000e]e1000_init_mac_params_82571 =_ "Please update your 82571 Bootagent\012"
drivers/net/ethernet/intel/e1000e/82571.c:548 [e1000e]e1000_get_hw_semaphore_82573 =_ "Driver can't access the PHY\012"
drivers/net/ethernet/intel/e1000e/82571.c:999 [e1000e]e1000_reset_hw_82571 =_ "Issuing a global reset to MAC\012"
drivers/net/ethernet/intel/e1000e/82571.c:971 [e1000e]e1000_reset_hw_82571 =_ "Masking off all interrupts\012"
drivers/net/ethernet/intel/e1000e/82571.c:969 [e1000e]e1000_reset_hw_82571 =_ "PCI-E Master disable polling has failed.\012"
drivers/net/ethernet/intel/e1000e/82571.c:1111 [e1000e]e1000_init_hw_82571 =_ "Zeroing the MTA\012"
drivers/net/ethernet/intel/e1000e/82571.c:1098 [e1000e]e1000_init_hw_82571 =_ "Initializing the IEEE VLAN\012"
drivers/net/ethernet/intel/e1000e/82571.c:1095 [e1000e]e1000_init_hw_82571 =_ "Error initializing identification LED\012"
drivers/net/ethernet/intel/e1000e/82571.c:875 [e1000e]e1000_get_cfg_done_82571 =_ "MNG configuration cycle has not completed.\012"
drivers/net/ethernet/intel/e1000e/82571.c:498 [e1000e]e1000_get_hw_semaphore_82571 =_ "Driver can't access the NVM\012"
drivers/net/ethernet/intel/e1000e/82571.c:480 [e1000e]e1000_get_hw_semaphore_82571 =_ "Driver can't access device - SMBI bit is set.\012"
drivers/net/ethernet/intel/e1000e/82571.c:1673 [e1000e]e1000_valid_led_default_82571 =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/82571.c:835 [e1000e]e1000_write_nvm_eewr_82571 =_ "nvm parameter(s) out of bounds\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3147 [e1000e]e1000_read_nvm_spt =_ "NVM read error: %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3091 [e1000e]e1000_read_nvm_spt =_ "Could not detect valid bank, assuming bank 0\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3082 [e1000e]e1000_read_nvm_spt =_ "nvm parameter(s) out of bounds\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4157 [e1000e]e1000_write_flash_data32_ich8lan =_ "Timeout error - flash cycle did not complete.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4203 [e1000e]e1000_retry_write_flash_dword_ich8lan =_ "Retrying Byte %8.8X at offset %u\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3538 [e1000e]e1000_read_flash_data32_ich8lan =_ "Timeout error - flash cycle did not complete.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3744 [e1000e]e1000_update_nvm_checksum_spt =_ "NVM update error: %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3681 [e1000e]e1000_update_nvm_checksum_spt =_ "Flash commit failed.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3615 [e1000e]e1000_update_nvm_checksum_spt =_ "Could not detect valid bank, assuming bank 0\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:512 [e1000e]e1000_init_phy_params_ich8lan =_ "Cannot determine PHY addr. Erroring out\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:598 [e1000e]e1000_init_nvm_params_ich8lan =_ "ERROR: Flash registers not mapped\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1760 [e1000e]e1000_rar_set_pch2lan =_ "Failed to write receive address at index %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1756 [e1000e]e1000_rar_set_pch2lan =_ "SHRA[%d] might be locked by ME - FWSM=0x%8.8x\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1870 [e1000e]e1000_rar_set_pch_lpt =_ "Failed to write receive address at index %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1024 [e1000e]e1000_platform_pm_pch_lpt =_ "Invalid LTR latency scale %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:999 [e1000e]e1000_platform_pm_pch_lpt =_ "Speed not set.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:993 [e1000e]e1000_platform_pm_pch_lpt =_ "max_frame_size not set.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1490 [e1000e]e1000_check_for_copper_link_ich8lan =_ "Error configuring flow control\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4552 [e1000e]e1000_reset_hw_ich8lan =_ "Issuing a global reset to ich8lan\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4502 [e1000e]e1000_reset_hw_ich8lan =_ "Masking off all interrupts\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4500 [e1000e]e1000_reset_hw_ich8lan =_ "PCI-E Master disable polling has failed.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4626 [e1000e]e1000_init_hw_ich8lan =_ "Zeroing the MTA\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4620 [e1000e]e1000_init_hw_ich8lan =_ "Error initializing identification LED\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4795 [e1000e]e1000_setup_link_ich8lan =_ "After fix-ups FlowControl is now = %x\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1625 [e1000e]e1000_acquire_swflag_ich8lan =_ "Failed to acquire the semaphore, FW or HW has it: FWSM=0x%8.8x EXTCNF_CTRL=0x%8.8x)\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1604 [e1000e]e1000_acquire_swflag_ich8lan =_ "SW has already locked the resource.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1590 [e1000e]e1000_acquire_swflag_ich8lan =_ "contention for Phy access\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:2682 [e1000e]e1000_lan_init_done_ich8lan =_ "LAN_INIT_DONE not set, increase timeout\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:5501 [e1000e]e1000_get_cfg_done_ich8lan =_ "EEPROM not present\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:5490 [e1000e]e1000_get_cfg_done_ich8lan =_ "PHY Reset Asserted not set - needs delay\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:5480 [e1000e]e1000_get_cfg_done_ich8lan =_ "Auto Read Done did not complete\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1656 [e1000e]e1000_release_swflag_ich8lan =_ "Semaphore unexpectedly released by sw/fw/hw\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3207 [e1000e]e1000_read_nvm_ich8lan =_ "NVM read error: %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3182 [e1000e]e1000_read_nvm_ich8lan =_ "Could not detect valid bank, assuming bank 0\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3173 [e1000e]e1000_read_nvm_ich8lan =_ "nvm parameter(s) out of bounds\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4073 [e1000e]e1000_write_flash_data_ich8lan =_ "Timeout error - flash cycle did not complete.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4235 [e1000e]e1000_retry_write_flash_byte_ich8lan =_ "Retrying Byte %2.2X at offset %u\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3284 [e1000e]e1000_flash_cycle_init_ich8lan =_ "Flash controller busy, cannot get access\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3228 [e1000e]e1000_flash_cycle_init_ich8lan =_ "Flash descriptor invalid.  SW Sequencing must be used.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3462 [e1000e]e1000_read_flash_data_ich8lan =_ "Timeout error - flash cycle did not complete.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3054 [e1000e]e1000_valid_nvm_bank_detect_ich8lan =_ "ERROR: No valid NVM bank present\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3025 [e1000e]e1000_valid_nvm_bank_detect_ich8lan =_ "Unable to determine valid NVM bank via EEC - reading flash signature\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3006 [e1000e]e1000_valid_nvm_bank_detect_ich8lan =_ "ERROR: No valid NVM bank present\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3895 [e1000e]e1000_update_nvm_checksum_ich8lan =_ "NVM update error: %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3845 [e1000e]e1000_update_nvm_checksum_ich8lan =_ "Flash commit failed.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3783 [e1000e]e1000_update_nvm_checksum_ich8lan =_ "Could not detect valid bank, assuming bank 0\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:4374 [e1000e]e1000_valid_led_default_ich8lan =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:3565 [e1000e]e1000_write_nvm_ich8lan =_ "nvm parameter(s) out of bounds\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1302 [e1000e]e1000_disable_ulp_lpt_lp =_ "Error in ULP disable flow: %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1215 [e1000e]e1000_disable_ulp_lpt_lp =_ "ULP_CONFIG_DONE cleared after %dmsec\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:353 [e1000e]e1000_init_phy_workarounds_pchlan =_ "Required LANPHYPC toggle blocked by ME\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:314 [e1000e]e1000_init_phy_workarounds_pchlan =_ "Failed to initialize PHY flow\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:5315 [e1000e]e1000_resume_workarounds_pchlan =_ "Error %d in resume workarounds\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:5285 [e1000e]e1000_resume_workarounds_pchlan =_ "Failed to setup iRST\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:5271 [e1000e]e1000_resume_workarounds_pchlan =_ "Failed to init PHY flow ret_val=%d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1925 [e1000e]e1000_write_smbus_addr =_ "Unsupported SMB frequency in PHY\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:5035 [e1000e]e1000e_set_kmrn_lock_loss_workaround_ich8lan =_ "Workaround applies to ICH8 only.\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1160 [e1000e]e1000_enable_ulp_lpt_lp =_ "Error in ULP enable flow: %d\012"
drivers/net/ethernet/intel/e1000e/ich8lan.c:1096 [e1000e]e1000_enable_ulp_lpt_lp =_ "CABLE_DISCONNECTED %s set after %dmsec\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:944 [e1000e]e1000_copper_link_setup_gg82563_80003es2lan =_ "Error Resetting the PHY\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:709 [e1000e]e1000_reset_hw_80003es2lan =_ "Issuing a global reset to MAC\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:694 [e1000e]e1000_reset_hw_80003es2lan =_ "Masking off all interrupts\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:692 [e1000e]e1000_reset_hw_80003es2lan =_ "PCI-E Master disable polling has failed.\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:762 [e1000e]e1000_init_hw_80003es2lan =_ "Zeroing the MTA\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:755 [e1000e]e1000_init_hw_80003es2lan =_ "Initializing the IEEE VLAN\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:752 [e1000e]e1000_init_hw_80003es2lan =_ "Error initializing identification LED\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:575 [e1000e]e1000_phy_force_speed_duplex_80003es2lan =_ "Waiting for forced speed/duplex link on GG82563 phy.\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:557 [e1000e]e1000_phy_force_speed_duplex_80003es2lan =_ "GG82563 PSCR: %X\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:525 [e1000e]e1000_get_cfg_done_80003es2lan =_ "MNG configuration cycle has not completed.\012"
drivers/net/ethernet/intel/e1000e/80003es2lan.c:316 [e1000e]e1000_acquire_swfw_sync_80003es2lan =_ "Driver can't access resource, SW_FW_SYNC timeout.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1776 [e1000e]e1000e_update_adaptive =_ "Not in Adaptive IFS mode!\012"
drivers/net/ethernet/intel/e1000e/mac.c:1750 [e1000e]e1000e_reset_adaptive =_ "Not in Adaptive IFS mode!\012"
drivers/net/ethernet/intel/e1000e/mac.c:1732 [e1000e]e1000e_disable_pcie_master =_ "Master requests are pending.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1469 [e1000e]e1000e_valid_led_default =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/mac.c:1448 [e1000e]e1000e_get_auto_rd_done =_ "Auto read by HW from NVM has not completed.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1408 [e1000e]e1000e_get_hw_semaphore =_ "Driver can't access the NVM\012"
drivers/net/ethernet/intel/e1000e/mac.c:1389 [e1000e]e1000e_get_hw_semaphore =_ "Driver can't access device - SMBI bit is set.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1343 [e1000e]e1000e_get_speed_and_duplex_copper =_ "%u Mbps, %s Duplex\012"
drivers/net/ethernet/intel/e1000e/mac.c:1306 [e1000e]e1000e_config_fc_after_link_up =_ "Error forcing flow control settings\012"
drivers/net/ethernet/intel/e1000e/mac.c:1294 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = NONE.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1288 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = Rx PAUSE frames only.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1274 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = Tx PAUSE frames only.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1259 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = Rx PAUSE frames only.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1256 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = FULL.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1200 [e1000e]e1000e_config_fc_after_link_up =_ "PCS Auto Neg has not completed.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1182 [e1000e]e1000e_config_fc_after_link_up =_ "Error forcing flow control settings\012"
drivers/net/ethernet/intel/e1000e/mac.c:1170 [e1000e]e1000e_config_fc_after_link_up =_ "Error getting link speed and duplex\012"
drivers/net/ethernet/intel/e1000e/mac.c:1161 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = NONE.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1155 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = Rx PAUSE frames only.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1141 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = Tx PAUSE frames only.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1126 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = Rx PAUSE frames only.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1123 [e1000e]e1000e_config_fc_after_link_up =_ "Flow Control = FULL.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1063 [e1000e]e1000e_config_fc_after_link_up =_ "Copper PHY and Auto Neg has not completed.\012"
drivers/net/ethernet/intel/e1000e/mac.c:1041 [e1000e]e1000e_config_fc_after_link_up =_ "Error forcing flow control settings\012"
drivers/net/ethernet/intel/e1000e/mac.c:1000 [e1000e]e1000e_force_mac_fc =_ "Flow control param set incorrectly\012"
drivers/net/ethernet/intel/e1000e/mac.c:982 [e1000e]e1000e_force_mac_fc =_ "hw->fc.current_mode = %u\012"
drivers/net/ethernet/intel/e1000e/mac.c:839 [e1000e]e1000_poll_fiber_serdes_link_generic =_ "Valid Link Found\012"
drivers/net/ethernet/intel/e1000e/mac.c:833 [e1000e]e1000_poll_fiber_serdes_link_generic =_ "Error while checking for link\012"
drivers/net/ethernet/intel/e1000e/mac.c:824 [e1000e]e1000_poll_fiber_serdes_link_generic =_ "Never got a valid link from auto-neg!!!\012"
drivers/net/ethernet/intel/e1000e/mac.c:788 [e1000e]e1000_commit_fc_settings_generic =_ "Flow control param set incorrectly\012"
drivers/net/ethernet/intel/e1000e/mac.c:888 [e1000e]e1000e_setup_fiber_serdes_link =_ "No signal detected\012"
drivers/net/ethernet/intel/e1000e/mac.c:874 [e1000e]e1000e_setup_fiber_serdes_link =_ "Auto-negotiation enabled\012"
drivers/net/ethernet/intel/e1000e/mac.c:662 [e1000e]e1000_set_default_fc_generic =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/mac.c:722 [e1000e]e1000e_setup_link_generic =_ "Initializing the Flow Control address, type and timer regs\012"
drivers/net/ethernet/intel/e1000e/mac.c:710 [e1000e]e1000e_setup_link_generic =_ "After fix-ups FlowControl is now = %x\012"
drivers/net/ethernet/intel/e1000e/mac.c:632 [e1000e]e1000e_check_for_serdes_link =_ "SERDES: Link down - autoneg failed\012"
drivers/net/ethernet/intel/e1000e/mac.c:628 [e1000e]e1000e_check_for_serdes_link =_ "SERDES: Link down - no sync.\012"
drivers/net/ethernet/intel/e1000e/mac.c:624 [e1000e]e1000e_check_for_serdes_link =_ "SERDES: Link down - invalid codewords detected in autoneg.\012"
drivers/net/ethernet/intel/e1000e/mac.c:621 [e1000e]e1000e_check_for_serdes_link =_ "SERDES: Link up - autoneg completed successfully.\012"
drivers/net/ethernet/intel/e1000e/mac.c:608 [e1000e]e1000e_check_for_serdes_link =_ "SERDES: Link down - force failed.\012"
drivers/net/ethernet/intel/e1000e/mac.c:604 [e1000e]e1000e_check_for_serdes_link =_ "SERDES: Link up - forced.\012"
drivers/net/ethernet/intel/e1000e/mac.c:588 [e1000e]e1000e_check_for_serdes_link =_ "Rx'ing /C/, enable AutoNeg and stop forcing link.\012"
drivers/net/ethernet/intel/e1000e/mac.c:579 [e1000e]e1000e_check_for_serdes_link =_ "Error configuring flow control\012"
drivers/net/ethernet/intel/e1000e/mac.c:566 [e1000e]e1000e_check_for_serdes_link =_ "NOT Rx'ing /C/, disable AutoNeg and force link.\012"
drivers/net/ethernet/intel/e1000e/mac.c:525 [e1000e]e1000e_check_for_fiber_link =_ "Rx'ing /C/, enable AutoNeg and stop forcing link.\012"
drivers/net/ethernet/intel/e1000e/mac.c:516 [e1000e]e1000e_check_for_fiber_link =_ "Error configuring flow control\012"
drivers/net/ethernet/intel/e1000e/mac.c:503 [e1000e]e1000e_check_for_fiber_link =_ "NOT Rx'ing /C/, disable AutoNeg and force link.\012"
drivers/net/ethernet/intel/e1000e/mac.c:465 [e1000e]e1000e_check_for_copper_link =_ "Error configuring flow control\012"
drivers/net/ethernet/intel/e1000e/mac.c:201 [e1000e]e1000_check_alt_mac_addr_generic =_ "Ignoring Alternate Mac Address with MC bit set\012"
drivers/net/ethernet/intel/e1000e/mac.c:191 [e1000e]e1000_check_alt_mac_addr_generic =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/mac.c:176 [e1000e]e1000_check_alt_mac_addr_generic =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/mac.c:141 [e1000e]e1000e_init_rx_addrs =_ "Clearing RAR[1-%u]\012"
drivers/net/ethernet/intel/e1000e/mac.c:136 [e1000e]e1000e_init_rx_addrs =_ "Programming MAC Address into RAR[0]\012"
drivers/net/ethernet/intel/e1000e/manage.c:81 [e1000e]e1000_mng_enable_host_if =_ "Previous command timeout failed.\012"
drivers/net/ethernet/intel/e1000e/manage.c:69 [e1000e]e1000_mng_enable_host_if =_ "E1000_HOST_EN bit disabled.\012"
drivers/net/ethernet/intel/e1000e/manage.c:62 [e1000e]e1000_mng_enable_host_if =_ "ARC subsystem not valid.\012"
drivers/net/ethernet/intel/e1000e/nvm.c:611 [e1000e]e1000e_update_nvm_checksum_generic =_ "NVM Write Error while updating checksum.\012"
drivers/net/ethernet/intel/e1000e/nvm.c:603 [e1000e]e1000e_update_nvm_checksum_generic =_ "NVM Read Error while updating checksum.\012"
drivers/net/ethernet/intel/e1000e/nvm.c:579 [e1000e]e1000e_validate_nvm_checksum_generic =_ "NVM Checksum Invalid\012"
drivers/net/ethernet/intel/e1000e/nvm.c:572 [e1000e]e1000e_validate_nvm_checksum_generic =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/nvm.c:516 [e1000e]e1000_read_pba_string_generic =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/nvm.c:505 [e1000e]e1000_read_pba_string_generic =_ "PBA string buffer too small\012"
drivers/net/ethernet/intel/e1000e/nvm.c:500 [e1000e]e1000_read_pba_string_generic =_ "NVM PBA number section invalid length\012"
drivers/net/ethernet/intel/e1000e/nvm.c:495 [e1000e]e1000_read_pba_string_generic =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/nvm.c:463 [e1000e]e1000_read_pba_string_generic =_ "PBA string buffer too small\012"
drivers/net/ethernet/intel/e1000e/nvm.c:459 [e1000e]e1000_read_pba_string_generic =_ "NVM PBA number is not stored as string\012"
drivers/net/ethernet/intel/e1000e/nvm.c:450 [e1000e]e1000_read_pba_string_generic =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/nvm.c:444 [e1000e]e1000_read_pba_string_generic =_ "NVM Read Error\012"
drivers/net/ethernet/intel/e1000e/nvm.c:438 [e1000e]e1000_read_pba_string_generic =_ "PBA string buffer was null\012"
drivers/net/ethernet/intel/e1000e/nvm.c:292 [e1000e]e1000_ready_nvm_eeprom =_ "SPI NVM Status error\012"
drivers/net/ethernet/intel/e1000e/nvm.c:364 [e1000e]e1000e_write_nvm_spi =_ "nvm parameter(s) out of bounds\012"
drivers/net/ethernet/intel/e1000e/nvm.c:331 [e1000e]e1000e_read_nvm_eerd =_ "NVM read error: %d\012"
drivers/net/ethernet/intel/e1000e/nvm.c:320 [e1000e]e1000e_read_nvm_eerd =_ "nvm parameter(s) out of bounds\012"
drivers/net/ethernet/intel/e1000e/nvm.c:187 [e1000e]e1000e_acquire_nvm =_ "Could not acquire NVM grant\012"
drivers/net/ethernet/intel/e1000e/phy.c:3166 [e1000e]e1000_get_phy_info_82577 =_ "Phy info is only valid if link is up\012"
drivers/net/ethernet/intel/e1000e/phy.c:3135 [e1000e]e1000_phy_force_speed_duplex_82577 =_ "Link taking longer than expected.\012"
drivers/net/ethernet/intel/e1000e/phy.c:3127 [e1000e]e1000_phy_force_speed_duplex_82577 =_ "Waiting for forced speed/duplex link on 82577 phy\012"
drivers/net/ethernet/intel/e1000e/phy.c:2915 [e1000e]__e1000_write_phy_reg_hv =_ "writing PHY page %d (or 0x%x shifted) reg 0x%x\012"
drivers/net/ethernet/intel/e1000e/phy.c:3024 [e1000e]e1000_access_phy_debug_regs_hv =_ "Could not access the Data port register\012"
drivers/net/ethernet/intel/e1000e/phy.c:3013 [e1000e]e1000_access_phy_debug_regs_hv =_ "Could not write the Address Offset port register\012"
drivers/net/ethernet/intel/e1000e/phy.c:2793 [e1000e]__e1000_read_phy_reg_hv =_ "reading PHY page %d (or 0x%x shifted) reg 0x%x\012"
drivers/net/ethernet/intel/e1000e/phy.c:2618 [e1000e]e1000_disable_phy_wakeup_reg_access_bm =_ "Could not restore PHY register %d.%d\012"
drivers/net/ethernet/intel/e1000e/phy.c:2610 [e1000e]e1000_disable_phy_wakeup_reg_access_bm =_ "Could not set Port Control page\012"
drivers/net/ethernet/intel/e1000e/phy.c:2582 [e1000e]e1000_enable_phy_wakeup_reg_access_bm =_ "Could not write PHY register %d.%d\012"
drivers/net/ethernet/intel/e1000e/phy.c:2568 [e1000e]e1000_enable_phy_wakeup_reg_access_bm =_ "Could not read PHY register %d.%d\012"
drivers/net/ethernet/intel/e1000e/phy.c:2561 [e1000e]e1000_enable_phy_wakeup_reg_access_bm =_ "Could not set Port Control page\012"
drivers/net/ethernet/intel/e1000e/phy.c:2691 [e1000e]e1000_access_phy_wakeup_reg_bm =_ "Could not access PHY reg %d.%d\012"
drivers/net/ethernet/intel/e1000e/phy.c:2676 [e1000e]e1000_access_phy_wakeup_reg_bm =_ "Could not write address opcode to page %d\012"
drivers/net/ethernet/intel/e1000e/phy.c:2671 [e1000e]e1000_access_phy_wakeup_reg_bm =_ "Accessing PHY page %d reg 0x%x\012"
drivers/net/ethernet/intel/e1000e/phy.c:2666 [e1000e]e1000_access_phy_wakeup_reg_bm =_ "Could not enable PHY wakeup reg access\012"
drivers/net/ethernet/intel/e1000e/phy.c:2660 [e1000e]e1000_access_phy_wakeup_reg_bm =_ "Attempting to access page %d while gig enabled.\012"
drivers/net/ethernet/intel/e1000e/phy.c:2157 [e1000e]e1000e_phy_init_script_igp3 =_ "Running IGP 3 PHY init script\012"
drivers/net/ethernet/intel/e1000e/phy.c:2034 [e1000e]e1000_get_phy_info_ife =_ "Phy info is only valid if link is up\012"
drivers/net/ethernet/intel/e1000e/phy.c:1976 [e1000e]e1000e_get_phy_info_igp =_ "Phy info is only valid if link is up\012"
drivers/net/ethernet/intel/e1000e/phy.c:1910 [e1000e]e1000e_get_phy_info_m88 =_ "Phy info is only valid if link is up\012"
drivers/net/ethernet/intel/e1000e/phy.c:1901 [e1000e]e1000e_get_phy_info_m88 =_ "Phy info is only valid for copper media\012"
drivers/net/ethernet/intel/e1000e/phy.c:1475 [e1000e]e1000e_phy_force_speed_duplex_setup =_ "Forcing 10mb\012"
drivers/net/ethernet/intel/e1000e/phy.c:1471 [e1000e]e1000e_phy_force_speed_duplex_setup =_ "Forcing 100mb\012"
drivers/net/ethernet/intel/e1000e/phy.c:1463 [e1000e]e1000e_phy_force_speed_duplex_setup =_ "Full Duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:1459 [e1000e]e1000e_phy_force_speed_duplex_setup =_ "Half Duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:1412 [e1000e]e1000_phy_force_speed_duplex_ife =_ "Link taking longer than expected.\012"
drivers/net/ethernet/intel/e1000e/phy.c:1404 [e1000e]e1000_phy_force_speed_duplex_ife =_ "Waiting for forced speed/duplex link on IFE phy.\012"
drivers/net/ethernet/intel/e1000e/phy.c:1399 [e1000e]e1000_phy_force_speed_duplex_ife =_ "IFE PMC: %X\012"
drivers/net/ethernet/intel/e1000e/phy.c:1311 [e1000e]e1000e_phy_force_speed_duplex_m88 =_ "Link taking longer than expected.\012"
drivers/net/ethernet/intel/e1000e/phy.c:1302 [e1000e]e1000e_phy_force_speed_duplex_m88 =_ "Waiting for forced speed/duplex link on M88 phy.\012"
drivers/net/ethernet/intel/e1000e/phy.c:1282 [e1000e]e1000e_phy_force_speed_duplex_m88 =_ "M88E1000 PSCR: %X\012"
drivers/net/ethernet/intel/e1000e/phy.c:1243 [e1000e]e1000e_phy_force_speed_duplex_igp =_ "Link taking longer than expected.\012"
drivers/net/ethernet/intel/e1000e/phy.c:1235 [e1000e]e1000e_phy_force_speed_duplex_igp =_ "Waiting for forced speed/duplex link on IGP phy.\012"
drivers/net/ethernet/intel/e1000e/phy.c:1230 [e1000e]e1000e_phy_force_speed_duplex_igp =_ "IGP PSCR: %X\012"
drivers/net/ethernet/intel/e1000e/phy.c:1069 [e1000e]e1000_phy_setup_autoneg =_ "Auto-Neg Advertising %x\012"
drivers/net/ethernet/intel/e1000e/phy.c:1061 [e1000e]e1000_phy_setup_autoneg =_ "Flow control param set incorrectly\012"
drivers/net/ethernet/intel/e1000e/phy.c:1004 [e1000e]e1000_phy_setup_autoneg =_ "Advertise 1000mb Full duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:1000 [e1000e]e1000_phy_setup_autoneg =_ "Advertise 1000mb Half duplex request denied!\012"
drivers/net/ethernet/intel/e1000e/phy.c:994 [e1000e]e1000_phy_setup_autoneg =_ "Advertise 100mb Full duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:988 [e1000e]e1000_phy_setup_autoneg =_ "Advertise 100mb Half duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:982 [e1000e]e1000_phy_setup_autoneg =_ "Advertise 10mb Full duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:976 [e1000e]e1000_phy_setup_autoneg =_ "Advertise 10mb Half duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:972 [e1000e]e1000_phy_setup_autoneg =_ "autoneg_advertised %x\012"
drivers/net/ethernet/intel/e1000e/phy.c:1129 [e1000e]e1000_copper_link_autoneg =_ "Error while waiting for autoneg to complete\012"
drivers/net/ethernet/intel/e1000e/phy.c:1109 [e1000e]e1000_copper_link_autoneg =_ "Restarting Auto-Neg\012"
drivers/net/ethernet/intel/e1000e/phy.c:1106 [e1000e]e1000_copper_link_autoneg =_ "Error Setting up Auto-Negotiation\012"
drivers/net/ethernet/intel/e1000e/phy.c:1103 [e1000e]e1000_copper_link_autoneg =_ "Reconfiguring auto-neg advertisement params\012"
drivers/net/ethernet/intel/e1000e/phy.c:1185 [e1000e]e1000e_setup_copper_link =_ "Unable to establish link!!!\012"
drivers/net/ethernet/intel/e1000e/phy.c:1181 [e1000e]e1000e_setup_copper_link =_ "Valid link established!!!\012"
drivers/net/ethernet/intel/e1000e/phy.c:1167 [e1000e]e1000e_setup_copper_link =_ "Error Forcing Speed and Duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:1164 [e1000e]e1000e_setup_copper_link =_ "Forcing Speed and Duplex\012"
drivers/net/ethernet/intel/e1000e/phy.c:863 [e1000e]e1000e_copper_link_setup_igp =_ "Error Disabling LPLU D0\012"
drivers/net/ethernet/intel/e1000e/phy.c:850 [e1000e]e1000e_copper_link_setup_igp =_ "Error resetting the PHY.\012"
drivers/net/ethernet/intel/e1000e/phy.c:814 [e1000e]e1000e_copper_link_setup_m88 =_ "Error committing the PHY changes\012"
drivers/net/ethernet/intel/e1000e/phy.c:758 [e1000e]e1000e_copper_link_setup_m88 =_ "Error committing the PHY changes\012"
drivers/net/ethernet/intel/e1000e/phy.c:316 [e1000e]e1000_set_page_igp =_ "Setting page 0x%x\012"
drivers/net/ethernet/intel/e1000e/phy.c:241 [e1000e]e1000e_write_phy_reg_mdic =_ "MDI Write offset error - requested %d, returned %d\012"
drivers/net/ethernet/intel/e1000e/phy.c:235 [e1000e]e1000e_write_phy_reg_mdic =_ "MDI Error\012"
drivers/net/ethernet/intel/e1000e/phy.c:231 [e1000e]e1000e_write_phy_reg_mdic =_ "MDI Write did not complete\012"
drivers/net/ethernet/intel/e1000e/phy.c:205 [e1000e]e1000e_write_phy_reg_mdic =_ "PHY Address %d is out of range\012"
drivers/net/ethernet/intel/e1000e/phy.c:177 [e1000e]e1000e_read_phy_reg_mdic =_ "MDI Read offset error - requested %d, returned %d\012"
drivers/net/ethernet/intel/e1000e/phy.c:171 [e1000e]e1000e_read_phy_reg_mdic =_ "MDI Error\012"
drivers/net/ethernet/intel/e1000e/phy.c:167 [e1000e]e1000e_read_phy_reg_mdic =_ "MDI Read did not complete\012"
drivers/net/ethernet/intel/e1000e/phy.c:142 [e1000e]e1000e_read_phy_reg_mdic =_ "PHY Address %d is out of range\012"
drivers/net/ethernet/intel/e1000e/netdev.c:4370 [e1000e]e1000_intr_msi_test =_ "icr is %08X\012"
drivers/net/ethernet/intel/e1000e/netdev.c:4438 [e1000e]e1000_test_msi_interrupt =_ "MSI interrupt test succeeded!\012"
drivers/net/ethernet/intel/e1000e/netdev.c:5048 [e1000e]e1000e_check_82574_phy_workaround =_ "PHY appears hung - resetting\012"
drivers/net/ethernet/intel/e1000e/netdev.c:1127 [e1000e]e1000_print_hw_hang =_ "false hang detected, ignoring\012"
drivers/net/ethernet/intel/e1000e/netdev.c:7167 [e1000e]e1000_probe =_ "NVM read error getting EEPROM version: %d\012"
drivers/net/ethernet/intel/e1000e/netdev.c:7144 [e1000e]e1000_probe =_ "NVM read error getting WoL initial values: %d\012"
drivers/net/ethernet/intel/e1000e/netdev.c:4242 [e1000e]e1000e_down =_ "failed to disable jumbo frame workaround mode\012"
drivers/net/ethernet/intel/e1000e/netdev.c:3054 [e1000e]e1000_setup_rctl =_ "failed to enable|disable jumbo frame workaround mode\012"
drivers/net/ethernet/intel/e1000e/netdev.c:982 [e1000e]e1000_clean_rx_irq =_ "Receive packet consumed multiple buffers\012"
drivers/net/ethernet/intel/e1000e/netdev.c:1375 [e1000e]e1000_clean_rx_irq_ps =_ "Last part of the packet spanning multiple descriptors\012"
drivers/net/ethernet/intel/e1000e/netdev.c:1359 [e1000e]e1000_clean_rx_irq_ps =_ "Packet Split buffers didn't pick up the full packet\012"
drivers/net/ethernet/intel/e1000e/ptp.c:176 [e1000e]e1000e_systim_overflow_work =_ "SYSTIM overflow check at %ld.%09lu\012"
Comment 16 Jarod Wilson 2016-02-03 21:43:59 UTC 
(In reply to HWSD from comment #15)
> Let me know if you need any specific debug.

The spew I was actually looking for was from dmesg when the  unstable link condition was occurring, with all that debug spew enabled. The control node is just showing *which* dynamic debug lines are enabled, not the ones that actually triggered while the problem was happening. This does confirm all the bonding and e1000e spew is enabled though, so hopefully, something useful shows up in dmesg. :)
Comment 17 HWSD 2016-02-04 08:07:09 UTC 
Indeed it seemed strange that there was need the control output...
But dmesg don't explain a lot....

I detached two network cables:
[53950.810085] e1000e: enp4s0 NIC Link is Down
[53950.862909] bond0: link status definitely down for interface enp4s0, disabling it
[54345.635938] e1000e: enp5s0 NIC Link is Down
[54345.681795] bond0: link status definitely down for interface enp5s0, disabling it
[54345.681805] bond0: now running without any active interface!
[54346.637894] LAN_HAW: port 1(bond0.1) entered disabled state
[54346.637979] DMZ: port 1(bond0.3) entered disabled state

Attached only one cable (network stable):
[54397.658131] e1000e: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[54397.671073] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[54397.671089] bond0: link status definitely up for interface enp4s0, 1000 Mbps full duplex
[54397.671094] bond0: first active interface up!
[54397.671244] LAN_HAW: port 1(bond0.1) entered forwarding state
[54397.671247] LAN_HAW: port 1(bond0.1) entered forwarding state
[54397.671257] DMZ: port 1(bond0.3) entered forwarding state
[54397.671259] DMZ: port 1(bond0.3) entered forwarding state

Attached both cables (netwok problems):
[54496.781735] e1000e: enp5s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[54496.850666] bond0: link status definitely up for interface enp5s0, 1000 Mbps full duplex
[54564.728023] hrtimer: interrupt took 10849 ns

Detached other cable (network stable):
[54653.102690] e1000e: enp4s0 NIC Link is Down
[54653.119516] bond0: link status definitely down for interface enp4s0, disabling it
[54653.119521] bond0: first active interface up!
Comment 21 HWSD 2016-02-04 17:16:25 UTC 
I installed a kernel update.
Bond still not working but dmesg is different.

[root@ovirt01 ~]# uname -a
Linux ovirt01 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

------------
dmesg

One cable attached

[ 2039.933105] e1000e 0000:04:00.0 enp4s0: hw->fc.current_mode = 0
[ 2039.933356] e1000e 0000:04:00.0 enp4s0: Flow Control = NONE.
[ 2039.933360] e1000e 0000:04:00.0 enp4s0: 1000 Mbps, Full Duplex
[ 2039.933364] e1000e 0000:04:00.0 enp4s0: hw->fc.current_mode = 0
[ 2039.933404] e1000e 0000:04:00.0 enp4s0: ARC subsystem not valid.
[ 2039.933921] e1000e 0000:04:00.0 enp4s0: 1000 Mbps, Full Duplex
[ 2039.933925] e1000e: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 2039.934288] e1000e 0000:04:00.0 enp4s0: event: 4
[ 2039.934289] e1000e 0000:04:00.0 enp4s0: IFF_SLAVE
[ 2039.934291] bond0: Port 1 Received link speed 0 update from adapter
[ 2039.934292] bond0: Port 1 changed speed
[ 2039.934293] bond0: Port 1 slave enp4s0 changed duplex
[ 2039.934295] bond_3ad_get_active_agg_info failed
[ 2040.002836] bond0: bond_should_notify_peers: slave NULL
[ 2040.002842] bond0: link status definitely up for interface enp4s0, 1000 Mbps full duplex
[ 2040.002845] bond0: Port 1 Received status full duplex update from adapter
[ 2040.002847] bond0: Port 1 Received link speed 4 update from adapter
[ 2040.002848] bond0: Port 1 changed link status to UP
[ 2040.002851] bond_3ad_get_active_agg_info failed
[ 2040.002857] bond0: now running without any active interface!
[ 2040.002862] Rx Machine: Port=1 (enp4s0), Last State=2, Curr State=1
[ 2040.002864] bond0: Port 1 left LAG 1
[ 2040.002866] LAG 1 was cleared
[ 2040.002868] bond0: Port 1 joined LAG 1(new LAG)
[ 2040.002870] bond0: best Agg=1; P=1; a k=9; p k=1; Ind=0; Act=0
[ 2040.002873] bond0: best ports ffff880036481038 slave ffff8804157c6a00 enp4s0
[ 2040.002875] bond0: Agg=1; P=1; a k=9; p k=1; Ind=0; Act=0
[ 2040.002877] bond0: Agg=2; P=1; a k=0; p k=1; Ind=1; Act=0
[ 2040.002878] bond0: LAG 1 chosen as the active LAG
[ 2040.002880] bond0: Agg=1; P=1; a k=9; p k=1; Ind=0; Act=1
[ 2040.002887] Mux Machine: Port=1 (enp4s0), Last State=3, Curr State=1
[ 2040.003012] bond0: event: 4
[ 2040.003012] bond0: IFF_MASTER
[ 2040.003105] bond0.1: event: 4
[ 2040.003109] LAN_HAW: port 1(bond0.1) entered forwarding state
[ 2040.003112] LAN_HAW: port 1(bond0.1) entered forwarding state
[ 2040.003121] bond0.3: event: 4
[ 2040.003123] DMZ: port 1(bond0.3) entered forwarding state
[ 2040.003125] DMZ: port 1(bond0.3) entered forwarding state
[ 2040.003132] LAN_HAW: event: 4
[ 2040.102782] bond0: bond_should_notify_peers: slave enp4s0
[ 2040.102789] Rx Machine: Port=1 (enp4s0), Last State=2, Curr State=4
[ 2040.102790] Periodic Machine: Port=1, Last State=1, Curr State=2
[ 2040.102792] Mux Machine: Port=1 (enp4s0), Last State=1, Curr State=2
[ 2040.202781] bond0: bond_should_notify_peers: slave enp4s0
[ 2040.202792] update lacpdu: enp4s0, actor port state c5
[ 2040.202800] Sent LACPDU on port 1
[ 2040.302752] bond0: bond_should_notify_peers: slave enp4s0
[ 2040.402726] bond0: bond_should_notify_peers: slave enp4s0
[ 2040.502696] bond0: bond_should_notify_peers: slave enp4s0
[ 2040.602695] bond0: bond_should_notify_peers: slave enp4s0
[ 2040.702672] bond0: bond_should_notify_peers: slave enp4s0
[ 2040.802651] bond0: bond_should_notify_peers: slave enp4s0
[ 2040.902630] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.002610] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.002618] Periodic Machine: Port=1, Last State=2, Curr State=4
[ 2041.102599] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.102606] Periodic Machine: Port=1, Last State=4, Curr State=2
[ 2041.102608] update lacpdu: enp4s0, actor port state c5
[ 2041.102613] Sent LACPDU on port 1
[ 2041.202568] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.302538] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.402526] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.502508] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.602486] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.702465] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.802444] bond0: bond_should_notify_peers: slave enp4s0
[ 2041.851975] e1000e 0000:05:00.0 enp5s0: ARC subsystem not valid.
[ 2041.902424] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.002406] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.002416] Periodic Machine: Port=1, Last State=2, Curr State=4
[ 2042.002419] update lacpdu: enp4s0, actor port state c5
[ 2042.002426] Sent LACPDU on port 1
[ 2042.102383] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.102391] Periodic Machine: Port=1, Last State=4, Curr State=2
[ 2042.102394] Mux Machine: Port=1 (enp4s0), Last State=2, Curr State=3
[ 2042.202370] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.302351] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.302363] update lacpdu: enp4s0, actor port state cd
[ 2042.302371] Sent LACPDU on port 1
[ 2042.402321] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.502286] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.602275] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.702243] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.802224] bond0: bond_should_notify_peers: slave enp4s0
[ 2042.902201] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.002196] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.002206] Periodic Machine: Port=1, Last State=2, Curr State=4
[ 2043.102179] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.102190] Rx Machine: Port=1 (enp4s0), Last State=4, Curr State=5
[ 2043.102193] Periodic Machine: Port=1, Last State=4, Curr State=3
[ 2043.202155] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.202167] update lacpdu: enp4s0, actor port state 4d
[ 2043.202175] Sent LACPDU on port 1
[ 2043.302142] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.402114] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.502092] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.602073] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.702050] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.802034] bond0: bond_should_notify_peers: slave enp4s0
[ 2043.851572] e1000e 0000:05:00.0 enp5s0: ARC subsystem not valid.

-----------------------
second cable attached:

[ 2049.943004] e1000e 0000:05:00.0 enp5s0: hw->fc.current_mode = 0
[ 2049.943255] e1000e 0000:05:00.0 enp5s0: Flow Control = NONE.
[ 2049.943259] e1000e 0000:05:00.0 enp5s0: 1000 Mbps, Full Duplex
[ 2049.943263] e1000e 0000:05:00.0 enp5s0: hw->fc.current_mode = 0
[ 2049.943302] e1000e 0000:05:00.0 enp5s0: ARC subsystem not valid.
[ 2049.943817] e1000e 0000:05:00.0 enp5s0: 1000 Mbps, Full Duplex
[ 2049.943822] e1000e: enp5s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 2049.944172] e1000e 0000:05:00.0 enp5s0: event: 4
[ 2049.944173] e1000e 0000:05:00.0 enp5s0: IFF_SLAVE
[ 2049.944175] bond0: Port 2 Received link speed 0 update from adapter
[ 2049.944176] bond0: Port 2 changed speed
[ 2049.944177] bond0: Port 2 slave enp5s0 changed duplex
[ 2050.000757] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.000764] bond0: link status definitely up for interface enp5s0, 1000 Mbps full duplex
[ 2050.000767] bond0: Port 2 Received status full duplex update from adapter
[ 2050.000769] bond0: Port 2 Received link speed 4 update from adapter
[ 2050.000771] bond0: Port 2 changed link status to UP
[ 2050.000780] Rx Machine: Port=2 (enp5s0), Last State=2, Curr State=1
[ 2050.000783] bond0: Port 2 left LAG 2
[ 2050.000785] LAG 2 was cleared
[ 2050.000787] bond0: Port 2 joined LAG 2(new LAG)
[ 2050.000789] bond0: Port 1 Received link speed 4 update from adapter
[ 2050.000791] bond0: Port 1 Received link speed 4 update from adapter
[ 2050.000792] bond0: Port 2 Received link speed 4 update from adapter
[ 2050.000794] bond0: Port 1 Received link speed 4 update from adapter
[ 2050.000796] Mux Machine: Port=2 (enp5s0), Last State=3, Curr State=1
[ 2050.100735] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.100744] Rx Machine: Port=2 (enp5s0), Last State=2, Curr State=4
[ 2050.100746] Periodic Machine: Port=2, Last State=1, Curr State=2
[ 2050.100748] Mux Machine: Port=2 (enp5s0), Last State=1, Curr State=2
[ 2050.100750] update lacpdu: enp5s0, actor port state c5
[ 2050.100757] Sent LACPDU on port 2
[ 2050.200714] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.300693] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.400669] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.500652] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.600636] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.700612] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.800597] bond0: bond_should_notify_peers: slave enp4s0
[ 2050.900572] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.000547] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.000556] Periodic Machine: Port=2, Last State=2, Curr State=4
[ 2051.000560] update lacpdu: enp5s0, actor port state c5
[ 2051.000568] Sent LACPDU on port 2
[ 2051.100524] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.100533] Periodic Machine: Port=2, Last State=4, Curr State=2
[ 2051.200510] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.300489] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.400470] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.500443] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.600420] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.700398] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.800380] bond0: bond_should_notify_peers: slave enp4s0
[ 2051.900358] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.000341] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.000349] Periodic Machine: Port=2, Last State=2, Curr State=4
[ 2052.100316] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.100324] Periodic Machine: Port=2, Last State=4, Curr State=2
[ 2052.100327] Mux Machine: Port=2 (enp5s0), Last State=2, Curr State=3
[ 2052.200296] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.200304] update lacpdu: enp5s0, actor port state c5
[ 2052.200311] Sent LACPDU on port 2
[ 2052.300278] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.400250] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.500225] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.600214] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.700189] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.800166] bond0: bond_should_notify_peers: slave enp4s0
[ 2052.900137] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.000131] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.000138] Periodic Machine: Port=2, Last State=2, Curr State=4
[ 2053.100098] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.100098] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.100105] Rx Machine: Port=2 (enp5s0), Last State=4, Curr State=5
[ 2053.100106] Periodic Machine: Port=2, Last State=4, Curr State=3
[ 2053.100107] update lacpdu: enp5s0, actor port state 45
[ 2053.100111] Sent LACPDU on port 2
[ 2053.200081] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.300060] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.400050] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.500035] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.600007] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.699986] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.799966] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.899931] bond0: bond_should_notify_peers: slave enp4s0
[ 2053.999924] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.099904] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.199870] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.299865] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.399843] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.499828] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.599802] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.699766] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.799757] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.899738] bond0: bond_should_notify_peers: slave enp4s0
[ 2054.999717] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.099698] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.199673] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.299656] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.399638] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.499602] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.599596] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.699570] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.799551] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.899534] bond0: bond_should_notify_peers: slave enp4s0
[ 2055.999515] bond0: bond_should_notify_peers: slave enp4s0
[ 2056.099495] bond0: bond_should_notify_peers: slave enp4s0
[ 2056.199466] bond0: bond_should_notify_peers: slave enp4s0
[ 2056.299451] bond0: bond_should_notify_peers: slave enp4s0
[ 2056.399437] bond0: bond_should_notify_peers: slave enp4s0
[ 2056.499396] bond0: bond_should_notify_peers: slave enp4s0
[ 2056.599370] bond0: bond_should_notify_peers: slave enp4s0
[ 2056.699374] bond0: bond_should_notify_peers: slave enp4s0
Comment 24 Jarod Wilson 2016-02-08 17:01:16 UTC 
(In reply to HWSD from comment #21)


> [ 2040.002868] bond0: Port 1 joined LAG 1(new LAG)
> [ 2040.002870] bond0: best Agg=1; P=1; a k=9; p k=1; Ind=0; Act=0
> [ 2040.002873] bond0: best ports ffff880036481038 slave ffff8804157c6a00
> enp4s0
> [ 2040.002875] bond0: Agg=1; P=1; a k=9; p k=1; Ind=0; Act=0
> [ 2040.002877] bond0: Agg=2; P=1; a k=0; p k=1; Ind=1; Act=0
> [ 2040.002878] bond0: LAG 1 chosen as the active LAG
> [ 2040.002880] bond0: Agg=1; P=1; a k=9; p k=1; Ind=0; Act=1


> -----------------------
> second cable attached:
> 
> [ 2049.943004] e1000e 0000:05:00.0 enp5s0: hw->fc.current_mode = 0
> [ 2049.943255] e1000e 0000:05:00.0 enp5s0: Flow Control = NONE.
> [ 2049.943259] e1000e 0000:05:00.0 enp5s0: 1000 Mbps, Full Duplex
> [ 2049.943263] e1000e 0000:05:00.0 enp5s0: hw->fc.current_mode = 0
> [ 2049.943302] e1000e 0000:05:00.0 enp5s0: ARC subsystem not valid.
> [ 2049.943817] e1000e 0000:05:00.0 enp5s0: 1000 Mbps, Full Duplex
> [ 2049.943822] e1000e: enp5s0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: None
> [ 2049.944172] e1000e 0000:05:00.0 enp5s0: event: 4
> [ 2049.944173] e1000e 0000:05:00.0 enp5s0: IFF_SLAVE
> [ 2049.944175] bond0: Port 2 Received link speed 0 update from adapter
> [ 2049.944176] bond0: Port 2 changed speed
> [ 2049.944177] bond0: Port 2 slave enp5s0 changed duplex
> [ 2050.000757] bond0: bond_should_notify_peers: slave enp4s0
> [ 2050.000764] bond0: link status definitely up for interface enp5s0, 1000
> Mbps full duplex
> [ 2050.000767] bond0: Port 2 Received status full duplex update from adapter
> [ 2050.000769] bond0: Port 2 Received link speed 4 update from adapter
> [ 2050.000771] bond0: Port 2 changed link status to UP
> [ 2050.000780] Rx Machine: Port=2 (enp5s0), Last State=2, Curr State=1
> [ 2050.000783] bond0: Port 2 left LAG 2
> [ 2050.000785] LAG 2 was cleared
> [ 2050.000787] bond0: Port 2 joined LAG 2(new LAG)

So there's definitely an issue here, which was actually illustrated in the initial report as well, but I neglected to notice. The two interfaces in the bond have managed to get themselves into different link aggregator groups (LAG 1 and LAG 2), which means they aren't aggregating with one another at all. Can you confirm that they were actually getting into the same LAG with the earlier kernel? I believe cat /proc/net/bonding/bond0 output from the functioning kernel should be enough to verify that. Not sure where to go next yet...
Comment 27 HWSD 2016-02-08 17:15:27 UTC 
(In reply to Jarod Wilson from comment #24)

> [CUT]
> Can you confirm that they were actually getting into the same LAG with
> the earlier kernel? I believe cat /proc/net/bonding/bond0 output from the
> functioning kernel should be enough to verify that. Not sure where to go
> next yet...

This info is already in the Comment Number 2:

Bond info when using kernel-3.10.0-229.20.1.el7.x86_64

[root@ovirt01 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 1
        Actor Key: 17
        Partner Key: 1
        Partner Mac Address: 00:00:00:00:00:00

Slave Interface: enp4s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: enp5s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 
Aggregator ID: 2
Slave queue ID: 0
Comment 28 Jarod Wilson 2016-02-08 17:42:42 UTC 
(In reply to HWSD from comment #27)
> (In reply to Jarod Wilson from comment #24)
> 
> > [CUT]
> > Can you confirm that they were actually getting into the same LAG with
> > the earlier kernel? I believe cat /proc/net/bonding/bond0 output from the
> > functioning kernel should be enough to verify that. Not sure where to go
> > next yet...
> 
> This info is already in the Comment Number 2:

D'oh, sorry, so it is.

> Bond info when using kernel-3.10.0-229.20.1.el7.x86_64
> 
> [root@ovirt01 ~]# cat /proc/net/bonding/bond0
> Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
> 
> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> Transmit Hash Policy: layer2 (0)
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 0
> Down Delay (ms): 0
> 
> 802.3ad info
> LACP rate: slow
> Min links: 0
> Aggregator selection policy (ad_select): stable
> Active Aggregator Info:
>         Aggregator ID: 1
>         Number of ports: 1
>         Actor Key: 17
>         Partner Key: 1
>         Partner Mac Address: 00:00:00:00:00:00
> 
> Slave Interface: enp4s0
> MII Status: up
> Speed: 1000 Mbps
> Duplex: full
> Link Failure Count: 0
> Permanent HW addr: 
> Aggregator ID: 1
> Slave queue ID: 0
> 
> Slave Interface: enp5s0
> MII Status: up
> Speed: 1000 Mbps
> Duplex: full
> Link Failure Count: 0
> Permanent HW addr: 
> Aggregator ID: 2
> Slave queue ID: 0

Your other setup is also not aggregating correctly, based on that output. enp4s0 has aggregator ID 1, enp5s0 has aggregator ID 2. The actual 802.3ad link reports aggregator ID 1, and only 1 "port" (which in the bonding driver is equivalent to an interface). I'm not sure what's actually changed that is causing traffic to misbehave now, but the link doesn't appear to have ever been actually doing what you'd intended, which makes it a little harder to debug...
Comment 29 HWSD 2016-02-08 17:55:50 UTC 
I have an old 10/100 switch. This may have LAG, I have to look into it. I will try this to isolate the problem on network cards or switch.

The mistery is why all worked fine for one year with kernel < 
kernel-3.10.0-327.3.1.el7.x86_64
Comment 30 Jarod Wilson 2016-02-08 21:31:52 UTC 
So as I understand it, the switch at the other end needs to support link aggregation, and probably needs to be explicitly configured such that it does 802.3ad lacp link aggregation on the two ports connected to the host. Basically, I believe this can only be used with a managed switch that you have configured appropriately. My own test setup actually shows the same thing, where the two ports have different LAG IDs, and I haven't got access to the switch configs to alter anything on that side.
Comment 31 HWSD 2016-02-08 21:40:07 UTC 
Switch support team (d-link) confirm me that the switch support 802.3ad lcap and the correct configuration made. 

I will try another switch. I also escalated d-link ticket to have other confirmaton on switch config and eventually firmware bugs.
Comment 32 HWSD 2016-02-08 21:40:37 UTC 
Switch support team (d-link) confirm me that the switch support 802.3ad lcap and the correct configuration made. 

I will try another switch. I also escalated d-link ticket to have other confirmaton on switch config and eventually firmware bugs.
Comment 35 HWSD 2016-02-09 08:29:17 UTC 
I made some test using an OLD Surecom EP-826DG-CS. Results are the same.
I configured two port trunking, two port each trunk. One trunk goes to d-link switch, other to the server.

The bond showes two differents LAG ID and the network link work only if there are only one network cable connected. The trunk between two switches work without problems.

Now I'm waiting reply from d-link support team.
Comment 36 Jarod Wilson 2016-02-09 14:14:48 UTC 
Is there a way to dump the settings for the trunking pairs on either of the switches, that might give some LACP/LAG/802.3ad information? There's also static trunking that can be configured on some switches, so I'm wondering if they're simply not talking the same link aggregation language at the moment.

Docs for your Surecom claim it has 802.3ad support, but the docs I found online are pretty slim on details for how to configure it... What model is the D-Link?
Comment 37 HWSD 2016-02-09 14:34:16 UTC 
Surecom switch hasn't a lot of settings... simply Port Trunking enabled or disabled in a list of preselected ports.

Also d-link has only the enabled or not option and port selection.

D-link switch is DGS-1100-08
Comment 38 Jarod Wilson 2016-02-10 17:37:06 UTC 
(In reply to HWSD from comment #37)
> Surecom switch hasn't a lot of settings... simply Port Trunking enabled or
> disabled in a list of preselected ports.
> 
> Also d-link has only the enabled or not option and port selection.
> 
> D-link switch is DGS-1100-08

According to this document on D-Link's web site...

http://www.dlink.com/-/media/Business_Products/DGS/DGS%201100/Datasheet/DGS_1100_Series_Datasheet_EN_EU.pdf

...the DGS-1100-08 does not appear to support 802.3ad link aggregation, only these ones do:

• 802.3ad Link Aggregation:
 • DGS-1100-16: Support max 8 groups per device and 8 ports per group
 • DGS-1100-10MP: Support 5 groups per device and 8 ports per group
 • DGS-1100-18: Support max 9 groups per device and 8 ports per group
 • DGS-1100-24P: Support max 12 groups per device and 8 ports per group
 • DGS-1100-26: Support max 13 groups per device and 8 ports per group
 • DGS-1100-26MP: Support 13 groups per device and 8 ports per group
Comment 39 HWSD 2016-02-15 13:49:12 UTC 
I got definitive aswer from d-link support team:

"DGS-1100-08 A1 does not support 802.3ad LACP"

The previous information given to me was wrong, then it is nitely a problem of the switch.
Comment 40 Jarod Wilson 2016-02-15 18:02:39 UTC 
(In reply to HWSD from comment #39)
> I got definitive aswer from d-link support team:
> 
> "DGS-1100-08 A1 does not support 802.3ad LACP"
> 
> The previous information given to me was wrong, then it is nitely a problem
> of the switch.

So should this bug be CLOSED->NOTABUG then, as it appears the root problem all along was the switch not supporting the bonding mode?
Comment 41 HWSD 2016-02-15 19:59:21 UTC 
Yes! Switch problem! It's strange that all worked until the upgrade. ... Bud d-link confirmed the switch isssue.
Comment 42 Jarod Wilson 2016-02-15 20:46:36 UTC 
(In reply to HWSD from comment #41)
> Yes! Switch problem! It's strange that all worked until the upgrade. ... Bud
> d-link confirmed the switch isssue.

Well, it only halfway worked, you weren't actually getting any link aggregation. :)

I'm still not sure why it stopped half-working, though I suspect tighter checking in the bonding driver makes the link not work when the other end isn't compliant. In any case, I think we just close this, as we can't really "fix" an 802.3ad link where one of the two endpoints isn't actually able to do 802.3ad.
Note You need to log in before you can comment on or make changes to this bug.