Created attachment 502027 [details]
dmesg output with NFS/NLM/RPC debug

btw. I'm reporting this issue to akpm@ and to Trond Myklebust because other vendors and upstream linux kernel is affected too.

Description of problem:
unprivileged user can use flock syscall for NFS share to trigger unlimited resource leak and this could cause node DoS.

Details:
flock on NFS calls nlmclnt_lock() function, where nlmclnt_call() executes synchronous RPC task NLMPROC_LOCK. Via several subcalls (call_bind) it successfully connects to NFS server and decodes its reply, however because server did not set up r_port rpcb_getport_done() returns -EACCES, and following rpc subcall call_bind_status() in this case repeats call_bind() after 3 sec timeout.

It isn't a big problem, and this cycle could be broken by signal, however error path of nlmclnt_lock() executes identical asynchronous RPC call to unlock taken lock. Waiting of result of this call is also canceled by signal, however in this case RPC call is asynchronous, it cycles too and never finished.

I would like to ask Trond to recheck my analyze and review my patch with fix of this issue or propose some better and more correct solution.

My idea is to break cycle described above, return -EOPNOTSUPP to nlmclnt_lock(), check it on error path and do not execute NLMPROC_UNLOCK call because we already know that server does not support locking operations.

PS. one more separate issue: NLMPROC_UNLOCK can be cycled in another place: nlmclnt_unlock_callback() restarts RPC task in case of any errors. Therefore we cannot execute NLMPROC_UNLOCK on error path in case of server errors. I did not reproduce this issue, however I believe it's possible.

PPS. Also I've found similar cycles in other nlm callbacks, however I have no ideas is it possible to exploit it. However I would like to ask Trond to review this code too.

Version-Release number of selected component (if applicable):
2.6.32-131.0.15.el6.x86_64
I've reproduced this issue on RHEL5, RHEL6, ubuntu 11.04 and upstream linux kernel 2.6.39. 

How reproducible:
100%

Actual results:
flock hangs, can be canceled by signal, as result one more rpc task leaks
(FYI: sysctl -w sunrpc.rpc_debug=0 dumps status of active RPC taks) 

Expected results:
no hangs, no leaks

Additional info:
dmesg output with NFS/NLM/RPC debug and concept of patch are attached.
Because  vendor-sec is no longer in use I'm preparing private report to akpm@ and to Trond Myklebust, probably they can propose some better solution