Bug 802040 - [f75277b12af4c7e72c448480feec0d0e2e6582cd] - dbench failed on nfs mount
[f75277b12af4c7e72c448480feec0d0e2e6582cd] - dbench failed on nfs mount
Status: CLOSED WORKSFORME
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
pre-release
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Rajesh
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-10 08:50 EST by M S Vishwanath Bhat
Modified: 2013-07-04 18:44 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-27 03:14:55 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
nfs logs (158.20 KB, text/x-log)
2012-03-10 08:50 EST, M S Vishwanath Bhat
no flags Details

  None (edit)
Description M S Vishwanath Bhat 2012-03-10 08:50:02 EST
Created attachment 569091 [details]
nfs logs

Description of problem:
Was running dbench on nfs mount. Cleaning up phased with Device or resource busy error.

Version-Release number of selected component (if applicable):
git master with head at f75277b12af4c7e72c448480feec0d0e2e6582cd

How reproducible:
2/2

Steps to Reproduce:
1. Create ans start a 2*2 distributed-replicated volume.
2. Now mount via nfs and run dbench for 600 secs with 30 clients.
  
Actual results:
  30     38469     8.99 MB/sec  execute 599 sec  latency 225.997 ms
  30  cleanup 600 sec
[36715] unlink './clients/client8/~dmtmp/WORD/.nfs48a15432497b15bf000000c3' failed - Device or resource busy
[36715] unlink './clients/client8/~dmtmp/WORD/.nfs5f177e8ceaa2489d000000c5' failed - Device or resource busy


Expected results:
dbench should not report error.

Additional info:

Entries from nfs log. 


[2012-03-10 06:16:12.649869] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.654532] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.833121] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.839149] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.962740] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:12.966558] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:14.523668] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:14.530925] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:14.721358] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:14.730868] W [fd-lk.c:428:print_lock_list] 0-fd-lk: lock list:
[2012-03-10 06:16:20.500305] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 7d51960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:20.952002] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 655960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.103481] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 5a56960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.148374] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: d656960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.223056] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: ab57960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.262496] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 1f58960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.277322] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 4c58960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.311498] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 9b58960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.336351] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: e958960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.350985] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 1d59960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.366202] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 4c59960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.377231] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 7c59960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.399424] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: b459960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.418445] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: df59960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.431437] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 45a960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.473666] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 795a960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.528196] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: fb5a960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.541833] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 1f5b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.552600] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 445b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.566166] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 645b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.575252] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 7c5b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.584440] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 905b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.594693] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: ab5b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.626557] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: fa5b960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.637099] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 125c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.645357] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 275c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.667796] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 565c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.673127] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 5c5c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.707362] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 975c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:16:21.714131] W [nfs3.c:3511:nfs3svc_rmdir_cbk] 0-nfs: 9d5c960f: /clients => -1 (Directory not empty)
[2012-03-10 06:23:28.540104] W [socket.c:1521:__socket_proto_state_machine] 0-NLM-client: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.11.112:46346)
[2012-03-10 06:23:28.540307] I [mem-pool.c:578:mem_pool_destroy] 0-nfs-server: size=2236 max=2 total=637
[2012-03-10 06:23:28.540338] I [mem-pool.c:578:mem_pool_destroy] 0-nfs-server: size=124 max=2 total=637


I have attached the nfs logs. Archived other logs.
Comment 1 Krishna Srinivas 2012-03-12 06:45:35 EDT
by making the client hostname resolvable in the server this problem was not seen anymore.
Can you see if the problem can still be reproduced?
Comment 2 M S Vishwanath Bhat 2012-03-12 07:35:28 EDT
(In reply to comment #1)
> by making the client hostname resolvable in the server this problem was not
> seen anymore.
> Can you see if the problem can still be reproduced?

The issue is not reproducible after after making client hostname resolvable from server.

Note You need to log in before you can comment on or make changes to this bug.