Bug 871968

Summary:	RPC tasks can deadlock during rpc_shutdown
Product:	Red Hat Enterprise Linux 6	Reporter:	Andy Adamson <andros>
Component:	kernel	Assignee:	Steve Dickson <steved>
Status:	CLOSED ERRATA	QA Contact:	Eryu Guan <eguan>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.4	CC:	dros, dwysocha, eguan, kzhang, steved, tlavigne
Target Milestone:	rc	Keywords:	Regression
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:	kernel-2.6.32-350.el6	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-02-21 06:54:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Andy Adamson 2012-10-31 21:16:28 UTC

Description of problem:
RPC tasks deadlock (randomly) when rpc_shutdown_client loops forever
in a workqueue kworker context, trying to kill all RPC tasks associated wit
the client, while one or more of these tasks have already been assigned to 
same kworker (and will never run rpc_exit_task).

Version-Release number of selected component (if applicable):


How reproducible:Very.

check /proc/fs/nfsfs/servers 

Steps to Reproduce:
1. Use server with WRITE delegations enabled (such as clustered ONTAP)

2. run nfsometer dd tests (http://wiki.linux-nfs.org/wiki/index.php/NFSometer)
this example is against clustered ONTAP:
nfsometer.py -m vers=4 -n 10 --admin-script=./nfsometerlib/scripts/server_ntap.sh 
--admin-server=<server> <server>:/<export> dd_100m_100k dd_100m_1k

3. the nfsometer run will go through the 10 steps w/out delegations then one of the first deleg_enabled steps should trigger an 
"Exception: NFS client not idle:" message.

4. After umount, /proc/fs/nfsfs/servers file should still show the server even
though umount 'succeeded'
  
Actual results: NFS umount leaves hung RPC tasks resulting in failed future mounts.


Expected results: Clean NFS umount.


Additional info:

These two upstream patches fix the bug.

commit 1b58f88c3f137875bff2ed382194ea56bda5fc9a
Author: Weston Andros Adamson <dros>
Date:   Tue Oct 30 17:01:40 2012 -0400

   SUNRPC: add WARN_ON_ONCE for potential deadlock

   rpc_shutdown_client should never be called from a workqueue context.
   If it is, it could deadlock looping forever trying to kill tasks that are
   assigned to the same kworker thread (and will never run rpc_exit_task).

   Signed-off-by: Weston Andros Adamson <dros>
   Signed-off-by: Trond Myklebust <Trond.Myklebust>

commit 324d003b0cd82151adbaecefef57b73f7959a469
Author: Weston Andros Adamson <dros>
Date:   Tue Oct 30 17:01:39 2012 -0400

   NFS: add nfs_sb_deactive_async to avoid deadlock

   Use nfs_sb_deactive_async instead of nfs_sb_deactive when in a workqueue
   context.  This avoids a deadlock where rpc_shutdown_client loops forever
   in a workqueue kworker context, trying to kill all RPC tasks associated wit
   the client, while one or more of these tasks have already been assigned to 
   same kworker (and will never run rpc_exit_task).

   This approach is needed because RPC tasks that have already been assigned
   to a kworker by queue_work cannot be canceled, as explained in the comment
   for workqueue.c:insert_wq_barrier.

   Signed-off-by: Weston Andros Adamson <dros>
   [Trond: add module_get/put.]
   Signed-off-by: Trond Myklebust <Trond.Myklebust>

Comment 2 Andy Adamson 2012-11-02 13:53:30 UTC

The first patch has a different commit hash:


commit 5ca0c0ed6082af0a9089403991d420bae58a555d
Author: Weston Andros Adamson <dros>
Date:   Tue Oct 30 17:01:40 2012 -0400

    SUNRPC: add WARN_ON_ONCE for potential deadlock
    
    rpc_shutdown_client should never be called from a workqueue context.
    If it is, it could deadlock looping forever trying to kill tasks that are
    assigned to the same kworker thread (and will never run rpc_exit_task).
    
    Signed-off-by: Weston Andros Adamson <dros>
    Signed-off-by: Trond Myklebust <Trond.Myklebust>

I would also like to point out that this bug is affecting RHEL 6.4 beta testing.

Comment 3 Weston Andros Adamson 2012-11-02 14:15:54 UTC

You don't need the "--admin-script=./nfsometerlib/scripts/server_ntap.sh 
--admin-server=<server>" options to nfsometer if the server has read and write delegations enabled.  The first run of a dd_* test using v4 will hit this deadlock.

You really don't need nfsometer either:

1) mount -o vers=4 server:/export /mnt
2) dd if=/dev/zero of=/mnt/ddfile bs=1024 count=102400 && umount /mnt
3) check /proc/fs/nfsfs/servers - the entry for "server" will never go away.  Also one of the kworker threads will be stuck in rpc_shutdown_client.

Note there are other ways to hit this deadlock - it's just that the delegreturn path is easier to reproduce.

Comment 5 RHEL Program Management 2012-12-12 19:29:48 UTC

This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.

Comment 8 Jarod Wilson 2012-12-20 21:25:11 UTC

Patch(es)

Comment 13 errata-xmlrpc 2013-02-21 06:54:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0496.html