Red Hat Bugzilla – Bug 138182
Kernel hang when cat'ting file on intr NFS mount
Last modified: 2007-11-30 17:07:04 EST
Description of problem:
Kernel hang when catting a file on an NFS share mounted with intr.
The problem occurs on x86 and x86_64. Backtraces attached below.
Version-Release number of selected component (if applicable):
2.4.21-20.EL (both UP and SMP)
works with 2.4.21-15.EL
Created attachment 106209 [details]
Created attachment 106210 [details]
appears LLNL is also seeing a similar problem. This is reported in IT
copying in info provided by LLNL in IT #54507:
The nfsbug.c reproducer will occaisionally reproduce the problem on
our machines. We think the key is that the last operation to have the
sillydeleted file open must be the completion of an asynchronous
write, so the reproducer attempts to write enough data to the deleted
nfs file so that the process closes the file before the writes
complete. This doesn't always happen, and I've found that ctrl-c-ing
out of "nfsbug" before the write loop completes can reproduce the
problem more regularly.
[in nfs dir]
This happens for me once every four or five times `nfsbug' is run.
I'll try to work on something that doesn't need to be killed in order
to reproduce the problem.
Created attachment 106965 [details]
After Mark told me about his reproducer, it makes sense to me how the
problem could be occurring.
When an unlink is done, the dentry count is checked. If the count is
> 1, some other process has the file open, so we shouldn't do an
unlink. A sillyrename is done instead and the flag
DCACHE_NFSFS_RENAMED is set on the dentry.
When the process with that file opened is done, it closes the file and
exits. Due to CTO consistency, all the dirty paged need to be synced
back to the nfs server. Rpc tasks are setup to do that through
nfs_file_flush() -> nfs_wb_all() -> nfs_sync_file() ->
nfs_flush_list() -> nfs_flush_one(). nfs_flush_one() sets up the rpc
task that calls nfs_writeback_done(). nfs_writeback_done(), when
called, will call nfs_dentry_iput(), which checks the flag
DCACHE_NFSFS_RENAMED, and if set, will call nfs_complete_unlink().
Now, nfs_flush_one() calls rpc_execute() on the task it just created,
instead of sticking it on the schedq. I don't know much about the rpc
scheduler, so if its possible for the task to be put to sleep and back
on the sched queue at some point, then rpciod would presumably
reschedule it later on, possibly calling nfs_complete_unlink() down
the road. I haven't been able to see it yet.
the problem is that nfs_unlink_complete runs in rpciod as an async
RPC task, but then calls wait_event. this makes rpciod sleep, which
prevents any other async RPC tasks from running.
the fix, i believe, is to make nfs_unlink_complete use the RPC
client's sleep primitives rather than nfs_wait_event. these should
do the right thing whether a sync or an async RPC task is calling
Created attachment 107280 [details]
LLNL was provided this patch from Trond via Chuck Lever
the fix idea was from trond, the patch is mine. i say that because
i'm still waiting for trond to review my work and post his version
(the official version) on the client.linux-nfs.org web site.
there are two ways to address this bug.
1. apply the patch i gave to LLNL, or
2. revert the fix-unlink patch.
the fix-unlink patch changes "rm -rf" on NFS to wait (or hang,
depending on your perspective) for open files to be closed. this is
more "unix-y" in that applications can continue to use unlinked files,
but it might be troublesome for folks using "rm -rf" on very large
fix #1 retains the nicer unix-like behavior, but fix #2 will make RHEL
3.0 behave more like earlier versions of Linux.
Pending Stephen Tweedie's code review and approval, I'll queue SteveD's
patch for the next U5 build (best case is tomorrow night). Personally,
I'm in favor of committing SteveD's new fix (as well as retaining his
original U3 fix), despite the slight divergence from Upstream NFS code
(which doesn't resolve the original problem fixed in Taroon U3).
A fix for this problem has just been committed to the RHEL3 U5
patch pool this evening (in kernel version 2.4.21-27.7.EL).
Changes to revert the U3 silly-delete and the U5 rpciod-hang patches have
just been committed to the RHEL3 U5 patch pool this afternoon (in kernel
Is the patch in attachment (id=107280) the latest version of the patch
that went into the U5 kernel 2.4.21-27.9.EL, or is there a more recent
Created attachment 111450 [details]
change committed to -27.9.EL
Jason, that's what was committed to -27.7.EL. In -27.9.EL, we reverted
that change plus the U3 patch that it was trying to fix. Attached above
is the patch that was committed to accomplish this.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.
*** Bug 139570 has been marked as a duplicate of this bug. ***
*** Bug 145550 has been marked as a duplicate of this bug. ***