Bug 127815 - (IT#36426) netconsole freezes during printk() when output link not up
netconsole freezes during printk() when output link not up
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
i686 Linux
medium Severity high
: ---
: ---
Assigned To: Jeff Moyer
Brian Brock
Depends On:
Blocks: 137214
  Show dependency treegraph
Reported: 2004-07-14 05:06 EDT by Alexandre Oliva
Modified: 2007-11-30 17:07 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2004-12-20 15:55:40 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
retry only n times on failed transmit due to queue_stopped (910 bytes, patch)
2004-07-28 11:41 EDT, Jeff Moyer
no flags Details | Diff
fixed version of the patch (950 bytes, patch)
2004-07-28 14:16 EDT, Jeff Moyer
no flags Details | Diff

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2004:550 normal SHIPPED_LIVE Updated kernel packages available for Red Hat Enterprise Linux 3 Update 4 2004-12-20 00:00:00 EST

  None (edit)
Description Alexandre Oliva 2004-07-14 05:06:12 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040510

Description of problem:
If you set up netconsole on a box with an e100 network card, unplug
the network cable for 20 seconds or so, then plug it back in, the
machine hangs right after printing the `e100: eth3 NIC Link is Up 100
Mbps Full duplex' message.

Version-Release number of selected component (if applicable):
kernel-2.4.21-15.0.3.EL netdump-0.6.11-3

How reproducible:

Steps to Reproduce:
1.Configure a machine with an e100 network card as a netconsole client
2.Unplug the network cable
3.Put it back in after 20 seconds

Actual Results:  Complete system freeze

Expected Results:  No freeze

Additional info:
Comment 2 Alexandre Oliva 2004-07-14 05:12:41 EDT
I have a strong suspicion that the problem has to do with the printk()
call in e100_watchdog() before e100_config_fc() and e100_config() are
called.  At that point, netif_running(dev) is already true, but the
interface is not configured so the interface polling from
write_netconsole_msg() may be unable to make any progress, so the
interface will never be configured.  Since the e100_watchdog() won't
be set up before the current execution is done, and interrupts are
disabled, we get a hard freeze.  I suspect moving the printk past the
e100_config() call might fix the problem.

It would still be possible to run into an error should any other
printk() be issued between the point when netif_running is set and
when the watchdog would kick in and reconfigures the interface.  We'd
get into the same deadlock, at least on uniprocessor machines.
Comment 4 Jeff Moyer 2004-07-20 16:24:19 EDT
I would like to see precisely where we are hung.  It would be helpful
to run this test on a system with the nmi_watchdog enabled.
Comment 6 Alexandre Oliva 2004-07-21 01:21:05 EDT
Jeffrey, we're stuck in the poll loop in write_console_msg(),
attempting to transmit the printk()ed string over the interface that
is not (re)configured yet, because it would only be configured when
printk returned.
Comment 7 John W. Linville 2004-07-21 11:15:41 EDT
Problem stems from netconsole handling output from printk()
synchronously.  While that may normally work, it does allow for this
sort of deadlock to occur.

netconsole either needs mechanism to handle printk() output
asynchrounously or it needs checks (e.g. netif_queue_stopped()) to
allow for dropping printk() when output device is unable to transmit.
Comment 11 Jeff Moyer 2004-07-26 19:04:19 EDT
If we change netconsole.c to discard the printk when
netif_queue_stopped returns true, then it is kind of a sledgehammer
approach.  So far, this thread has only concentrated on this one
reason that the queue is stopped.  Other reasons include running out
of TX descriptors, which can be fixed in the polling loop where we are
currently looping forever.  IOW, in other cases, calling the poll
function for the driver will free up these resources, and we will be
able to send out the printk.

I spoke with Matt Mackall at OLS, and he mentioned that deadlocks such
as this were considered, and the decision was made to handle the
issues on a case by case basis.  (well, he was actually talking about
a deadlock involving taking a spin lock in the interrupt routine, and
doing a printk with it held).

What this really comes down to is how reliable is the network console
expected to be?  There are certainly cases where we will have to drop
messages, and since the messages are packaged up using UDP, I guess we
aren't exactly advertising any guarantees.

If we agree on that, then I'll put together a patch which addresses
this in netconsole.  It may be enough to simply try to poll a few
times, and if we don't make any progress, return failure.
Comment 12 John W. Linville 2004-07-27 09:19:06 EDT
FWIW, I would agree that netconsole's reliability is at most "best
effort".  Dropping text output while the link goes up and down, while
undesirable, seems reasonable -- much more so than hanging the box... :-)
Comment 13 Alexandre Oliva 2004-07-28 04:18:31 EDT
Agreed.  After the argument that packets are sent with UDP, dropping
them in cases we can't transmit doesn't feel inappropriate at all.
Comment 14 Jeff Moyer 2004-07-28 11:41:36 EDT
Created attachment 102255 [details]
retry only n times on failed transmit due to queue_stopped

Here is a patch which implements my last suggestion.  It has been tested with
numerous cable pulls using the e100 driver.  This also makes things happy in
the case where you generate a lot of console output while the cable is
unplugged (which was also capable of hanging the machine).
Comment 15 Jeff Moyer 2004-07-28 14:16:51 EDT
Created attachment 102259 [details]
fixed version of the patch

I generated this patch before I saved my emacs buffer.	Good thing I tested it
Comment 16 Ernie Petrides 2004-09-03 20:49:14 EDT
A fix for this problem has just been committed to the RHEL3 U4
patch pool this evening (in kernel version 2.4.21-20.3.EL).
Comment 22 John Flanagan 2004-12-20 15:55:40 EST
An errata has been issued which should help the problem 
described in this bug report. This report is therefore being 
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, 
please follow the link below. You may reopen this bug report 
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.