Bug 526888 - NFSv4 reclaimer thread in an infinite loop
Summary: NFSv4 reclaimer thread in an infinite loop
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.4
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Sachin Prabhu
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks: 526950 529162
TreeView+ depends on / blocked
 
Reported: 2009-10-02 11:31 UTC by Sachin Prabhu
Modified: 2018-10-20 04:15 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-03-30 06:58:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
proposed fix (582 bytes, patch)
2009-10-02 11:44 UTC, Sachin Prabhu
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2010:0178 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 5.5 kernel security and bug fix update 2010-03-29 12:18:21 UTC

Description Sachin Prabhu 2009-10-02 11:31:36 UTC
A bug in nfs4_do_open_expired() can lead to the reclaimer thread going into an  infinite loop.

The stack traces appear similar to the one below seen on a 2.6.18-164 kernel.

192.168.23.73 R  running task       0  5372    155                2979 (L-TLB)    
ffff81042b68dc60 ffffffff80063024 0000000000000000 ffffffff80064ca1
0000000000402100 0000000000000001 ffff81043cdb6100 ffffffff802ffae0
000003965552e8df 0000000000003abc ffff81043cdb62e8 0000000000000000
Call Trace:
[<ffffffff80063024>] thread_return+0xbe/0xfe
[<ffffffff80064ca1>] __reacquire_kernel_lock+0x2c/0x45
[<ffffffff884b249c>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
[<ffffffff8853c19a>] :nfs:decode_compound_hdr+0x3d/0x82
[<ffffffff884af7f1>] :sunrpc:xprt_timer+0x0/0x7f
[<ffffffff8853c070>] :nfs:decode_op_hdr+0xd/0x5f
[<ffffffff8003da91>] lock_timer_base+0x1b/0x3c
[<ffffffff8002e2f2>] __wake_up+0x38/0x4f
[<ffffffff8852505e>] :nfs:nfs_access_get_cached+0x24/0x108
[<ffffffff8853a58e>] :nfs:_nfs4_do_access+0x2d/0x85
[<ffffffff8853b314>] :nfs:nfs4_open_expired+0x6c/0x16c
[<ffffffff8009f4a9>] keventd_create_kthread+0x0/0xc4
[<ffffffff88542359>] :nfs:nfs4_reclaim_open_state+0x2d/0x150
[<ffffffff88542620>] :nfs:reclaimer+0x1a4/0x2ac
[<ffffffff8854247c>] :nfs:reclaimer+0x0/0x2ac
[<ffffffff8003298b>] kthread+0xfe/0x132
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff8009f4a9>] keventd_create_kthread+0x0/0xc4
[<ffffffff8003288d>] kthread+0x0/0x132
[<ffffffff8005dfa7>] child_rip+0x0/0x11

The bug in nfs4_do_open_expired() is triggered if the client receives NFS4ERR_DELAY from the server. The exception.retry is set and a timeout enforced. However when the server recovers and returns the desired value, the exception.retry is never reset to 0 leading to an loop in the do..while loop. This was confirmed with an instrumented kernel.

This issue has been reported upstream and fixed in the commit
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=027b6ca02192f381a5a91237ba8a8cf625dc6f6a

Comment 5 RHEL Program Management 2009-10-05 17:19:04 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 10 Don Zickus 2009-10-13 16:09:43 UTC
in kernel-2.6.18-169.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please do NOT transition this bugzilla state to VERIFIED until our QE team
has sent specific instructions indicating when to do so.  However feel free
to provide a comment indicating that this fix has been verified.

Comment 18 Simone Caldana 2010-03-18 11:00:19 UTC
I am using kernel-2.6.18-191 but this happens regularly in the following setup:

Solaris 10 NFS server
Centos 5.4 Linux client 
relevant /etc/fstab line:
server:/storage nfs4 bg,intr,nosuid,actimeo=2,rsize=8192,wsize=8192

the low actimeo it's because the server changes metadata that the clients must
see asap.

Comment 19 Sachin Prabhu 2010-03-18 14:55:22 UTC
Simone,

You may be seeing a different problem.

To confirm that you see this problem, the next time you see the hang
echo w >/proc/sysrq-trigger 
a few times with a second's interval between each call. You need to check for a process which has the name set to the ip address of the nfs server. The back trace seen should be similar to the one seen in c#1.

Sachin Prabhu

Comment 20 Simone Caldana 2010-03-18 15:12:14 UTC
Sachin, 

I can already confirm the process presence. I can't exactly say about the stack trace now, I'll try to get one if and when the problem arises again (we stopped using that architecture for the critical projects and reversed using an older way of doing things). The machine slowly fades into what effectively is an hang (while it is not, because it pings)

Comment 22 errata-xmlrpc 2010-03-30 06:58:21 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0178.html


Note You need to log in before you can comment on or make changes to this bug.