Bug 526888

Summary: NFSv4 reclaimer thread in an infinite loop
Product: Red Hat Enterprise Linux 5 Reporter: Sachin Prabhu <sprabhu>
Component: kernelAssignee: Sachin Prabhu <sprabhu>
Status: CLOSED ERRATA QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.4CC: aleksey, dhoward, dzickus, jlayton, jpirko, jtluka, marcus, rwheeler, simone, steved, tao, wezhang
Target Milestone: rcKeywords: OtherQA, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-03-30 06:58:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 526950, 529162    
Attachments:
Description Flags
proposed fix none

Description Sachin Prabhu 2009-10-02 11:31:36 UTC
A bug in nfs4_do_open_expired() can lead to the reclaimer thread going into an  infinite loop.

The stack traces appear similar to the one below seen on a 2.6.18-164 kernel.

192.168.23.73 R  running task       0  5372    155                2979 (L-TLB)    
ffff81042b68dc60 ffffffff80063024 0000000000000000 ffffffff80064ca1
0000000000402100 0000000000000001 ffff81043cdb6100 ffffffff802ffae0
000003965552e8df 0000000000003abc ffff81043cdb62e8 0000000000000000
Call Trace:
[<ffffffff80063024>] thread_return+0xbe/0xfe
[<ffffffff80064ca1>] __reacquire_kernel_lock+0x2c/0x45
[<ffffffff884b249c>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
[<ffffffff8853c19a>] :nfs:decode_compound_hdr+0x3d/0x82
[<ffffffff884af7f1>] :sunrpc:xprt_timer+0x0/0x7f
[<ffffffff8853c070>] :nfs:decode_op_hdr+0xd/0x5f
[<ffffffff8003da91>] lock_timer_base+0x1b/0x3c
[<ffffffff8002e2f2>] __wake_up+0x38/0x4f
[<ffffffff8852505e>] :nfs:nfs_access_get_cached+0x24/0x108
[<ffffffff8853a58e>] :nfs:_nfs4_do_access+0x2d/0x85
[<ffffffff8853b314>] :nfs:nfs4_open_expired+0x6c/0x16c
[<ffffffff8009f4a9>] keventd_create_kthread+0x0/0xc4
[<ffffffff88542359>] :nfs:nfs4_reclaim_open_state+0x2d/0x150
[<ffffffff88542620>] :nfs:reclaimer+0x1a4/0x2ac
[<ffffffff8854247c>] :nfs:reclaimer+0x0/0x2ac
[<ffffffff8003298b>] kthread+0xfe/0x132
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff8009f4a9>] keventd_create_kthread+0x0/0xc4
[<ffffffff8003288d>] kthread+0x0/0x132
[<ffffffff8005dfa7>] child_rip+0x0/0x11

The bug in nfs4_do_open_expired() is triggered if the client receives NFS4ERR_DELAY from the server. The exception.retry is set and a timeout enforced. However when the server recovers and returns the desired value, the exception.retry is never reset to 0 leading to an loop in the do..while loop. This was confirmed with an instrumented kernel.

This issue has been reported upstream and fixed in the commit
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=027b6ca02192f381a5a91237ba8a8cf625dc6f6a

Comment 5 RHEL Program Management 2009-10-05 17:19:04 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 10 Don Zickus 2009-10-13 16:09:43 UTC
in kernel-2.6.18-169.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please do NOT transition this bugzilla state to VERIFIED until our QE team
has sent specific instructions indicating when to do so.  However feel free
to provide a comment indicating that this fix has been verified.

Comment 18 Simone Caldana 2010-03-18 11:00:19 UTC
I am using kernel-2.6.18-191 but this happens regularly in the following setup:

Solaris 10 NFS server
Centos 5.4 Linux client 
relevant /etc/fstab line:
server:/storage nfs4 bg,intr,nosuid,actimeo=2,rsize=8192,wsize=8192

the low actimeo it's because the server changes metadata that the clients must
see asap.

Comment 19 Sachin Prabhu 2010-03-18 14:55:22 UTC
Simone,

You may be seeing a different problem.

To confirm that you see this problem, the next time you see the hang
echo w >/proc/sysrq-trigger 
a few times with a second's interval between each call. You need to check for a process which has the name set to the ip address of the nfs server. The back trace seen should be similar to the one seen in c#1.

Sachin Prabhu

Comment 20 Simone Caldana 2010-03-18 15:12:14 UTC
Sachin, 

I can already confirm the process presence. I can't exactly say about the stack trace now, I'll try to get one if and when the problem arises again (we stopped using that architecture for the critical projects and reversed using an older way of doing things). The machine slowly fades into what effectively is an hang (while it is not, because it pings)

Comment 22 errata-xmlrpc 2010-03-30 06:58:21 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0178.html