Bug 614559

All,
Just to be clear...

This issue occurs only with NetDump enabled.
/kernel/networking/ndnc enables NetDump.

Testing with /kernel/filesystems/nfs/connectathon/
alone does reproduced issue.

Best,
-pbunyan

All,
One of the test jobs for this issue PANIC'd the system:

Running the /kernel/networking/ndnc and /kernel/filesystems/nfs/connectathon/ with
the 2.6.9-89.29.EL kernel the phenom-01.lab.bos.redhat.com PANIC'd.


Job#166463  Recipe-427100
http://rhts.redhat.com/cgi-bin/rhts/jobs.cgi?id=166463
x86_64 - kernel 2.6.9-89.29.EL
	/kernel/filesystems/nfs/connectathon
	Recipe-427100
  	 Test - /kernel/filesystems/nfs/connectathon/ - PANIC
	  http://rhts.redhat.com/testlogs/2010/07/166463/427100/3492852/console.txt

<-SNIP->
07/14/10 10:34:57  JobID:166463 Test:/kernel/filesystems/nfs/connectathon Response:1
07/14/10 10:34:58  testID:3492852 start:
[-- MARK -- Wed Jul 14 10:35:00 2010]
lockd: can't encode arguments: 5
<-SNIP->
[-- MARK -- Wed Jul 14 11:00:00 2010]
sky2 eth1: tx timeout
net/core/netpoll.c:300: spin_trylock(net/core/dev.c:000001022d311898) already locked by net/sched/sch_generic.c/189
net/core/netpoll.c:300: spin_trylock(net/core/dev.c:000001022d311898) already locked by net/sched/sch_generic.c/189
<-SNIP->
[-- MARK -- Wed Jul 14 12:30:00 2010]
sky2 eth1: tx timeout
sky2 eth1: tx timeout
[-- MARK -- Wed Jul 14 12:35:00 2010]
Badness in dst_release at include/net/dst.h:158

Call Trace:<IRQ> <ffffffff802fba08>{__kfree_skb+98} <ffffffffa00f330e>{:sky2:sky2_tx_complete+374} 
       <ffffffffa00f4f00>{:sky2:sky2_poll+2522} <ffffffff80302753>{net_rx_action+305} 
       <ffffffff80140c90>{__do_softirq+76} <ffffffff80140d17>{do_softirq+49} 
       <ffffffff80113f1f>{do_IRQ+664} <ffffffff8011110f>{ret_from_intr+0} 
        <EOI> <ffffffff802b6770>{ide_outsw+0} <ffffffff8010e807>{default_idle+0} 
       <ffffffff8010e827>{default_idle+32} <ffffffff8010e897>{cpu_idle+26} 
       <ffffffff80575704>{start_kernel+637} <ffffffff805751b7>{_sinittext+439} 
       
Badness in dst_release at include/net/dst.h:158

Call Trace:<IRQ> <ffffffff802fba08>{__kfree_skb+98} <ffffffffa00f330e>{:sky2:sky2_tx_complete+374} 
       <ffffffffa00f4f00>{:sky2:sky2_poll+2522} <ffffffff80302753>{net_rx_action+305} 
       <ffffffff80140c90>{__do_softirq+76} <ffffffff80140d17>{do_softirq+49} 
       <ffffffff80113f1f>{do_IRQ+664} <ffffffff8011110f>{ret_from_intr+0} 
        <EOI> <ffffffff802b6770>{ide_outsw+0} <ffffffff8010e807>{default_idle+0} 
       <ffffffff8010e827>{default_idle+32} <ffffffff8010e897>{cpu_idle+26} 
       <ffffffff80575704>{start_kernel+637} <ffffffff805751b7>{_sinittext+439} 
       
Unable to handle kernel paging request at 0000000102e93a7c RIP: 


Best,
-pbunyan

This last backtrace shows an issue with dst_release.  Specificallly it indicates that the skbs being freed had an underflowed refcount on the dst entry attached to them, which suggests that someone is calling dst_release more often than dst_hold in some path.  My guess would be that sky2_tx_timeout is racing with sky2_tx_done in interrupt context on another cpu and we're double freeing skbs on on the tx_cons list in sky2.  Thats just a symptom of the larger problem though.  while that can be fixed, it still doesnt' explain why sky2 is getting tx timeouts in the first place.

A scratch build that included the patch here: 
https://bugzilla.kernel.org/show_bug.cgi?id=6839#c55
has successfully passed cthon testing.

Looks like that was a backport of this:

commit 6771290102c4703dae56bc3e121deb63530e206c
Author: Stephen Hemminger <shemminger>
Date:   Mon Dec 4 15:53:45 2006 -0800

    [PATCH] sky2: beter ram buffer partitioning

Correction it looks like this is the patch we are looking for:

commit 470ea7eba4aaa517533f9b02ac9a104e77264548
Author: Stephen Hemminger <shemminger>
Date:   Fri Oct 20 17:06:11 2006 -0700

    [PATCH] sky2: 88E803X transmit lockup
    
and the patch in comment #7 came after it.

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

I built a test kernel with sky2 patch built in (comment 9). Paul still noticed the issue with sky2 driver. Following is small part of error message.

What attracts my attention here is first message of "pci hw error". Does it mean that there was some kind of hardware error which froze the transmission on card and then rest of the errors just followed...

***************************************************************************
sky2 0000:03:00.0: pci hw error (0x2010)
sky2 eth1: tx timeout
net/core/netpoll.c:300: spin_trylock(net/core/dev.c:000001022d01d898) already locked by net/sched/sch_generic.c/189
net/core/netpoll.c:300: spin_trylock(net/core/dev.c:000001022d01d898) already locked by net/sched/sch_generic.c/189
net/core/netpoll.c:300: spin_trylock(net/core/dev.c:000001022d01d898) already locked by net/sched/sch_generic.c/189
net/core/netpoll.c:300: spin_trylock(net/core/dev.c:000001022d01d898) already locked by net/sched/sch_generic.c/189
net/core/netpoll.c:300: spin_trylock(net/core/dev.c:000001022d01d898) already locked by net/sched/sch_generic.c/189
sky2 eth1: tx timeout
[-- MARK -- Wed Jul 21 12:10:00 2010]
sky2 eth1: tx timeout
sky2 eth1: tx timeout
nfs: server rhel5-nfs not responding, still trying
sky2 eth1: tx timeout
nfs: server rhel5-nfs not responding, still trying
sky2 eth1: tx timeout
******************************************************************************

[Testing update from PaulB]

All,

Just to follow up and add to the puzzle...
JeffB and I tested a bit more.

Testing with latest Z-Stream and Y-Stream kernels on phenom-01 system
with /kernel/networking/ndnc and /kernel/filesystems/nfs/connectathon
is intermittent.


However, testing with latest Z-Stream and Y-Stream kernels on amd-mako-02
system with /kernel/networking/ndnc and /kernel/filesystems/nfs/connectathon
we ran into a further glitch.
The issue with amd-mako-02 during testing was as follows:
[a] The system would successfully install the RHEL4u8 base kernel.
[b] However, after rebooting the system would not get an ip. Thus
    the system could not check in to continue the test. The system would
    hang and the test would eventually watchdog.
[c] We did hook up a crash cart to the system to follow the issue.
    As the system was in this state, following the install of RHEL4u8
    and reboot. Logging in to the systems and
    $service network stop
    $modprobe -r sky2
    $modprobe sky2
    $service network start
    $ifconfig -a
    The system was then able to get an ip.
    Similar to comment#18 in
    https://bugzilla.redhat.com/show_bug.cgi?id=216799
    Note: simply stopping an restarting network service was not
    successful. Success was only achieved following removing and
    reinserting the sky2 module

Further.
The amd-mako-02 system is connected to the network via a Cisco Catalyst
2960G 10/100/1000 switch. We modified this and connected the the amd-mako-02
to a NetGear FS608 10/100 switch, and the NetGear to the Cisco. I then reran
the same testing. The system would successfully install the RHEL4u8 base
kernel and _now_ following reboot successfully get an ip.
Howver, I did experience the same intermittent failure as before running
latest Z-Stream and Y-Stream kernel with /kernel/networking/ndnc
and /kernel/filesystems/nfs/connectathon.

Committed in 89.30.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/

I think I might know whats going on here.  This caught my eye in the summary:
sky2 0000:03:00.0: pci hw error (0x2010)

rhts isn't being responsive, so I can't confirm that precedes the other errors as well, but if it does, I think this doesn't have anything to do with netdump, but more specifically netconsole.  The hardware error path in sky2 is full of printks that are rate limited.  So if we get a hardware error at the wrong time, we'll send a whole bunch of printks down the tx path of the sky2 driver while we're in the middle of trying to recover from a major error (for the sake of recording it, the above is a  PCI MASTER ABORT)

I'll attach a patch shortly.

Created attachment 472682 [details]
patch to prevent printks on netconsole in sky2

This isn't necessecarily a permanent fix, but I think it should solve the probelm.  It will disable the netdevices queue, causing us to drop frames in dev_queue_xmit, thereby preventing us from accessing the tx path of the sky2 driver while trying to recover from a hw error.

Please give this patch a test and see if it remedies the problem.  Thanks!

well, it seems as though we have a marginal improvement.  We no longer see the lockdep warnings that arise from trying to use netconsole while we're recovering from a tx hw error.   Now we need to figure out why it is that the sky2 driver can't reset itself properly.  I'll dig into that now

Committed in 97.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
When running with the following Ethernet device:
 Ethernet controller: Marvell Technology Group Ltd. 
 88E8052 PCI-E ASF Gigabit Ethernet Controller (rev 20)
 Vendor ID:11ab
 Device ID:4360

Netconsole must be disabled to avoid this issue.

Please note:
   When the "sky2 eth1: tx timeout" issue occurs, the system unresponsive. I am unable to login to the system tty terminals or serial console. Therefore, I am unable to try to rmmod/insmod or ifdown/ifup. I must reboot the system to recover from this issue.

Updating "Technical Note"...

-pbunyan

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -4,4 +4,5 @@
  Vendor ID:11ab
  Device ID:4360
 
-Netconsole must be disabled to avoid this issue.+Disabling netconsole does _not_ resolve this issue.
+System must be rebooted to recover from this issue.

(In reply to comment #29)
> Please note:
>    When the "sky2 eth1: tx timeout" issue occurs, the system unresponsive. I am
> unable to login to the system tty terminals or serial console. Therefore, I am
> unable to try to rmmod/insmod or ifdown/ifup. I must reboot the system to
> recover from this issue.
> 
> Updating "Technical Note"...
> 
> -pbunyan

What seems odd about the tx timeout issues is that we don't see any "NETDEV WATCHDOG" messages in the log before we see the tx timeout messages from the driver.

Those messages should come out as KERN_INFO (6) messages shouldn't be coming out.  If /proc/sys/kernel/printk is "7 4 1 7" then we should see them.

I also wonder if we should try to backport this fix:

commit 819067916d785cac0369b8d6e187b4a83fd17785
Author: Stephen Hemminger <shemminger>
Date:   Thu Feb 15 16:40:33 2007 -0800

    sky2: transmit timeout
    
    The transmit timeout code could hang, and it would not clear out
    problems if the hardware was stuck.  Change the code to effectively do
    a device down/up similar to the suspend/resume code.

(In reply to comment #33)
> I also wonder if we should try to backport this fix:
> 
> commit 819067916d785cac0369b8d6e187b4a83fd17785
> Author: Stephen Hemminger <shemminger>
> Date:   Thu Feb 15 16:40:33 2007 -0800
> 
>     sky2: transmit timeout
> 
>     The transmit timeout code could hang, and it would not clear out
>     problems if the hardware was stuck.  Change the code to effectively do
>     a device down/up similar to the suspend/resume code.

Neil horman backported that patch and we tried it. It did not help.

(In reply to comment #34)
> (In reply to comment #33)
> > I also wonder if we should try to backport this fix:
> > 
> > commit 819067916d785cac0369b8d6e187b4a83fd17785
> > Author: Stephen Hemminger <shemminger>
> > Date:   Thu Feb 15 16:40:33 2007 -0800
> > 
> >     sky2: transmit timeout
> > 
> >     The transmit timeout code could hang, and it would not clear out
> >     problems if the hardware was stuck.  Change the code to effectively do
> >     a device down/up similar to the suspend/resume code.
> 
> Neil horman backported that patch and we tried it. It did not help.

I knew he tried something, but no longer had the emails with that information.  Thanks for the update, Vivek!

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,8 +1,18 @@
-When running with the following Ethernet device:
+[Silas: the following issue needs a CCFR description: what was the nature of the issue? Could a Subject Matter Expert provide sentences about each of the below to be edited and incorporated into the RHEL 4.9 Tech Notes?
+
+Cause: 
+
+Consequence: 
+
+Fix:
+
+Result: 
+
+Original: When running with the following Ethernet device:
  Ethernet controller: Marvell Technology Group Ltd. 
  88E8052 PCI-E ASF Gigabit Ethernet Controller (rev 20)
  Vendor ID:11ab
  Device ID:4360
 
 Disabling netconsole does _not_ resolve this issue.
-System must be rebooted to recover from this issue.+System must be rebooted to recover from this issue].

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,18 +1,12 @@
-[Silas: the following issue needs a CCFR description: what was the nature of the issue? Could a Subject Matter Expert provide sentences about each of the below to be edited and incorporated into the RHEL 4.9 Tech Notes?
+When using the sky2 driver for Marvell Ethernet adapters with the following device:
 
-Cause: 
+    Ethernet controller: Marvell Technology Group Ltd.
+    88E8052 PCI-E ASF Gigabit Ethernet Controller (rev 20)
+    Vendor ID:11ab
+    Device ID:4360
 
-Consequence: 
+packet transmission may time out with the following message being written to netconsole:
 
-Fix:
+    sky2 eth1: tx timeout
 
-Result: 
+This is a known issue: after receiving these messages, the system must be rebooted in order to fix the packet transmission issues. (BZ#614559)-
-Original: When running with the following Ethernet device:
- Ethernet controller: Marvell Technology Group Ltd. 
- 88E8052 PCI-E ASF Gigabit Ethernet Controller (rev 20)
- Vendor ID:11ab
- Device ID:4360
-
-Disabling netconsole does _not_ resolve this issue.
-System must be rebooted to recover from this issue].

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0263.html