A bug in nfs4_do_open_expired() can lead to the reclaimer thread going into an infinite loop. The stack traces appear similar to the one below seen on a 2.6.18-164 kernel. 192.168.23.73 R running task 0 5372 155 2979 (L-TLB) ffff81042b68dc60 ffffffff80063024 0000000000000000 ffffffff80064ca1 0000000000402100 0000000000000001 ffff81043cdb6100 ffffffff802ffae0 000003965552e8df 0000000000003abc ffff81043cdb62e8 0000000000000000 Call Trace: [<ffffffff80063024>] thread_return+0xbe/0xfe [<ffffffff80064ca1>] __reacquire_kernel_lock+0x2c/0x45 [<ffffffff884b249c>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28 [<ffffffff8853c19a>] :nfs:decode_compound_hdr+0x3d/0x82 [<ffffffff884af7f1>] :sunrpc:xprt_timer+0x0/0x7f [<ffffffff8853c070>] :nfs:decode_op_hdr+0xd/0x5f [<ffffffff8003da91>] lock_timer_base+0x1b/0x3c [<ffffffff8002e2f2>] __wake_up+0x38/0x4f [<ffffffff8852505e>] :nfs:nfs_access_get_cached+0x24/0x108 [<ffffffff8853a58e>] :nfs:_nfs4_do_access+0x2d/0x85 [<ffffffff8853b314>] :nfs:nfs4_open_expired+0x6c/0x16c [<ffffffff8009f4a9>] keventd_create_kthread+0x0/0xc4 [<ffffffff88542359>] :nfs:nfs4_reclaim_open_state+0x2d/0x150 [<ffffffff88542620>] :nfs:reclaimer+0x1a4/0x2ac [<ffffffff8854247c>] :nfs:reclaimer+0x0/0x2ac [<ffffffff8003298b>] kthread+0xfe/0x132 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff8009f4a9>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003288d>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11 The bug in nfs4_do_open_expired() is triggered if the client receives NFS4ERR_DELAY from the server. The exception.retry is set and a timeout enforced. However when the server recovers and returns the desired value, the exception.retry is never reset to 0 leading to an loop in the do..while loop. This was confirmed with an instrumented kernel. This issue has been reported upstream and fixed in the commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=027b6ca02192f381a5a91237ba8a8cf625dc6f6a
Created attachment 363463 [details] proposed fix Patch backported from http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=027b6ca02192f381a5a91237ba8a8cf625dc6f6a
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
in kernel-2.6.18-169.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please do NOT transition this bugzilla state to VERIFIED until our QE team has sent specific instructions indicating when to do so. However feel free to provide a comment indicating that this fix has been verified.
I am using kernel-2.6.18-191 but this happens regularly in the following setup: Solaris 10 NFS server Centos 5.4 Linux client relevant /etc/fstab line: server:/storage nfs4 bg,intr,nosuid,actimeo=2,rsize=8192,wsize=8192 the low actimeo it's because the server changes metadata that the clients must see asap.
Simone, You may be seeing a different problem. To confirm that you see this problem, the next time you see the hang echo w >/proc/sysrq-trigger a few times with a second's interval between each call. You need to check for a process which has the name set to the ip address of the nfs server. The back trace seen should be similar to the one seen in c#1. Sachin Prabhu
Sachin, I can already confirm the process presence. I can't exactly say about the stack trace now, I'll try to get one if and when the problem arises again (we stopped using that architecture for the critical projects and reversed using an older way of doing things). The machine slowly fades into what effectively is an hang (while it is not, because it pings)
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0178.html