Bug 138182 - Kernel hang when cat'ting file on intr NFS mount
Summary: Kernel hang when cat'ting file on intr NFS mount
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel
Version: 3.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Steve Dickson
QA Contact:
URL:
Whiteboard:
: 139570 145550 (view as bug list)
Depends On:
Blocks: 132991
TreeView+ depends on / blocked
 
Reported: 2004-11-05 11:50 UTC by Bastien Nocera
Modified: 2007-11-30 22:07 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2005-05-18 13:28:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
nfshang-x86_64-sysrq-t.txt (70.84 KB, text/plain)
2004-11-05 11:51 UTC, Bastien Nocera
no flags Details
nfshang-x86-sysrq-t.txt (102.71 KB, text/plain)
2004-11-05 11:51 UTC, Bastien Nocera
no flags Details
reproducer (1.12 KB, application/octet-stream)
2004-11-18 16:09 UTC, Dave Maley
no flags Details
unlink-deadlock patch (2.07 KB, patch)
2004-11-23 03:22 UTC, Dave Maley
no flags Details | Diff
change committed to -27.9.EL (2.99 KB, patch)
2005-02-26 00:10 UTC, Ernie Petrides
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2005:294 0 normal SHIPPED_LIVE Moderate: Updated kernel packages available for Red Hat Enterprise Linux 3 Update 5 2005-05-18 04:00:00 UTC

Description Bastien Nocera 2004-11-05 11:50:35 UTC
Description of problem:
Kernel hang when catting a file on an NFS share mounted with intr.
The problem occurs on x86 and x86_64. Backtraces attached below.

Version-Release number of selected component (if applicable):
2.4.21-20.EL (both UP and SMP)
works with 2.4.21-15.EL

Comment 1 Bastien Nocera 2004-11-05 11:51:05 UTC
Created attachment 106209 [details]
nfshang-x86_64-sysrq-t.txt

Comment 2 Bastien Nocera 2004-11-05 11:51:38 UTC
Created attachment 106210 [details]
nfshang-x86-sysrq-t.txt

Comment 6 Dave Maley 2004-11-17 21:17:03 UTC
appears LLNL is also seeing a similar problem.  This is reported in IT
#54507.

Comment 7 Dave Maley 2004-11-18 16:08:59 UTC
copying in info provided by LLNL in IT #54507:
--------------------------------------------------

The nfsbug.c reproducer will occaisionally reproduce the problem on
our machines. We think the key is that the last operation to have the
sillydeleted file open must be the completion of an asynchronous
write, so the reproducer attempts to write enough data to the deleted
nfs file so that the process closes the file before the writes
complete. This doesn't always happen, and I've found that ctrl-c-ing
out of "nfsbug" before the write loop completes can reproduce the
problem more regularly.

e.g.:

[in nfs dir]

./nfsbug
[hit ctrl-c]
hang

This happens for me once every four or five times `nfsbug' is run.
I'll try to work on something that doesn't need to be killed in order
to reproduce the problem.

Comment 8 Dave Maley 2004-11-18 16:09:50 UTC
Created attachment 106965 [details]
reproducer

Comment 9 Dave Maley 2004-11-18 16:11:45 UTC
After Mark told me about his reproducer, it makes sense to me how the
problem could be occurring.

When an unlink is done, the dentry count is checked.  If the count is
> 1, some other process has the file open, so we shouldn't do an
unlink.  A sillyrename is done instead and the flag
DCACHE_NFSFS_RENAMED is set on the dentry.

When the process with that file opened is done, it closes the file and
exits.  Due to CTO consistency, all the dirty paged need to be synced
back to the nfs server.  Rpc tasks are setup to do that through
nfs_file_flush() -> nfs_wb_all() -> nfs_sync_file() ->
nfs_flush_list() -> nfs_flush_one().  nfs_flush_one() sets up the rpc
task that calls nfs_writeback_done().  nfs_writeback_done(), when
called, will call nfs_dentry_iput(), which checks the flag
DCACHE_NFSFS_RENAMED, and if set, will call nfs_complete_unlink().

Now, nfs_flush_one() calls rpc_execute() on the task it just created,
instead of sticking it on the schedq.  I don't know much about the rpc
scheduler, so if its possible for the task to be put to sleep and back
on the sched queue at some point, then rpciod would presumably
reschedule it later on, possibly calling nfs_complete_unlink() down
the road.  I haven't been able to see it yet.

Al

Comment 10 Chuck Lever 2004-11-18 16:26:05 UTC
the problem is that nfs_unlink_complete runs in rpciod as an async 
RPC task, but then calls wait_event.  this makes rpciod sleep, which 
prevents any other async RPC tasks from running.

the fix, i believe, is to make nfs_unlink_complete use the RPC 
client's sleep primitives rather than nfs_wait_event.  these should 
do the right thing whether a sync or an async RPC task is calling 
nfs_unlink_complete.

Comment 11 Dave Maley 2004-11-23 03:22:42 UTC
Created attachment 107280 [details]
unlink-deadlock patch

LLNL was provided this patch from Trond via Chuck Lever

Comment 12 Chuck Lever 2004-11-23 03:42:34 UTC
the fix idea was from trond, the patch is mine.  i say that because
i'm still waiting for trond to review my work and post his version
(the official version) on the client.linux-nfs.org web site.

Comment 15 Chuck Lever 2004-11-29 21:50:54 UTC
there are two ways to address this bug.

1.  apply the patch i gave to LLNL, or

2.  revert the fix-unlink patch.

the fix-unlink patch changes "rm -rf" on NFS to wait (or hang,
depending on your perspective) for open files to be closed.  this is
more "unix-y" in that applications can continue to use unlinked files,
but it might be troublesome for folks using "rm -rf" on very large
directory trees.

fix #1 retains the nicer unix-like behavior, but fix #2 will make RHEL
3.0 behave more like earlier versions of Linux.

Comment 28 Ernie Petrides 2005-01-06 21:44:40 UTC
Pending Stephen Tweedie's code review and approval, I'll queue SteveD's
patch for the next U5 build (best case is tomorrow night).  Personally,
I'm in favor of committing SteveD's new fix (as well as retaining his
original U3 fix), despite the slight divergence from Upstream NFS code
(which doesn't resolve the original problem fixed in Taroon U3).


Comment 31 Ernie Petrides 2005-01-11 23:49:09 UTC
A fix for this problem has just been committed to the RHEL3 U5
patch pool this evening (in kernel version 2.4.21-27.7.EL).


Comment 38 Ernie Petrides 2005-01-25 23:39:08 UTC
Changes to revert the U3 silly-delete and the U5 rpciod-hang patches have
just been committed to the RHEL3 U5 patch pool this afternoon (in kernel
version 2.4.21-27.9.EL).


Comment 46 Jason Smith 2005-02-10 15:43:57 UTC
Is the patch in attachment (id=107280) the latest version of the patch
that went into the U5 kernel 2.4.21-27.9.EL, or is there a more recent
patch?


Comment 47 Ernie Petrides 2005-02-26 00:10:14 UTC
Created attachment 111450 [details]
change committed to -27.9.EL

Jason, that's what was committed to -27.7.EL.  In -27.9.EL, we reverted
that change plus the U3 patch that it was trying to fix.  Attached above
is the patch that was committed to accomplish this.

Comment 48 Tim Powers 2005-05-18 13:28:25 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2005-294.html


Comment 49 Ernie Petrides 2005-09-08 00:01:05 UTC
*** Bug 139570 has been marked as a duplicate of this bug. ***

Comment 50 Ernie Petrides 2005-09-08 00:07:37 UTC
*** Bug 139570 has been marked as a duplicate of this bug. ***

Comment 51 Ernie Petrides 2005-10-04 20:18:51 UTC
*** Bug 145550 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.