Created attachment 569091 [details]
nfs logs

Description of problem:
Was running dbench on nfs mount. Cleaning up phased with Device or resource busy error.

Version-Release number of selected component (if applicable):
git master with head at f75277b12af4c7e72c448480feec0d0e2e6582cd

How reproducible:
2/2

Steps to Reproduce:
1. Create ans start a 2*2 distributed-replicated volume.
2. Now mount via nfs and run dbench for 600 secs with 30 clients.
  
Actual results:
  30     38469     8.99 MB/sec  execute 599 sec  latency 225.997 ms
  30  cleanup 600 sec
[36715] unlink './clients/client8/~dmtmp/WORD/.nfs48a15432497b15bf000000c3' failed - Device or resource busy
[36715] unlink './clients/client8/~dmtmp/WORD/.nfs5f177e8ceaa2489d000000c5' failed - Device or resource busy


Expected results:
dbench should not report error.

Additional info:

Entries from nfs log. 


[2012-03-10 06:16:12.649869] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.654532] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.833121] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.839149] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.962740] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.966558] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:14.523668] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:14.530925] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:14.721358] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:14.730868] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:20.500305] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 7d51960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:20.952002] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 655960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.103481] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 5a56960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.148374] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: d656960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.223056] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: ab57960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.262496] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 1f58960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.277322] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 4c58960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.311498] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 9b58960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.336351] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: e958960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.350985] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 1d59960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.366202] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 4c59960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.377231] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 7c59960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.399424] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: b459960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.418445] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: df59960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.431437] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 45a960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.473666] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 795a960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.528196] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: fb5a960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.541833] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 1f5b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.552600] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 445b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.566166] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 645b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.575252] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 7c5b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.584440] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 905b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.594693] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: ab5b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.626557] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: fa5b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.637099] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 125c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.645357] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 275c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.667796] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 565c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.673127] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 5c5c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.707362] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 975c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.714131] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 9d5c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:23:28.540104] W [socket.c:1521:__socket_proto_state_machine] 0-NLM-client: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.11.112:46346)
[2012-03-10 06:23:28.540307] I [mem-pool.c:578:mem_pool_destroy] 0-nfs-server: size=2236 max=2 total=637
[2012-03-10 06:23:28.540338] I [mem-pool.c:578:mem_pool_destroy] 0-nfs-server: size=124 max=2 total=637


I have attached the nfs logs. Archived other logs.