Attachment 915750 Details for Bug 999399 – Comment

Comment

comment_6232170.txt (text/plain), 77.69 KB, created by Rachana Patel on 2013-08-21 09:20:41 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Rachana Patel

Created: 2013-08-21 09:20:41 UTC

Size: 77.69 KB

patch

obsolete

>Description of problem:
>When volume is full , ls on nfs mount is not showing any data and FUSE mount shows 2 entries for the same entity
>
>Version-Release number of selected component (if applicable):
>3.4.0.20rhs-2.el6rhs.x86_64
>
>How reproducible:
>not tried
>
>Steps to Reproduce:
>1. had a dist-rep volume(3x2) as below
>
>[root@DVM1 ~]# gluster v info master1
> 
>Volume Name: master1
>Type: Distributed-Replicate
>Volume ID: fa11e206-d039-4606-92fa-29f29a9a8dfa
>Status: Started
>Number of Bricks: 3 x 2 = 6
>Transport-type: tcp
>Bricks:
>Brick1: 10.70.37.128:/rhs/brick1
>Brick2: 10.70.37.110:/rhs/brick1
>Brick3: 10.70.37.192:/rhs/brick1
>Brick4: 10.70.37.88:/rhs/brick1
>Brick5: 10.70.37.81:/rhs/brick1
>Brick6: 10.70.37.88:/rhs/brick5/2
>Options Reconfigured:
>diagnostics.client-log-level: TRACE
>geo-replication.ignore-pid-check: on
>geo-replication.indexing: on
>changelog.encoding: ascii
>changelog.rollover-time: 15
>changelog.fsync-interval: 3
>[root@DVM1 ~]# gluster v status master1
>Status of volume: master1
>Gluster process						Port	Online	Pid
>------------------------------------------------------------------------------
>Brick 10.70.37.128:/rhs/brick1				49152	Y	14019
>Brick 10.70.37.110:/rhs/brick1				49152	Y	11927
>Brick 10.70.37.192:/rhs/brick1				49152	Y	11868
>Brick 10.70.37.88:/rhs/brick1				49152	Y	12242
>Brick 10.70.37.81:/rhs/brick1				49152	Y	12462
>Brick 10.70.37.88:/rhs/brick5/2				49153	Y	12253
>NFS Server on localhost					2049	Y	24100
>Self-heal Daemon on localhost				N/A	Y	14047
>NFS Server on 10.70.37.110				2049	Y	21488
>Self-heal Daemon on 10.70.37.110			N/A	Y	11946
>NFS Server on 10.70.37.81				2049	Y	21946
>Self-heal Daemon on 10.70.37.81				N/A	Y	12481
>NFS Server on 10.70.37.192				2049	Y	21462
>Self-heal Daemon on 10.70.37.192			N/A	Y	11887
>NFS Server on 10.70.37.88				2049	Y	22049
>Self-heal Daemon on 10.70.37.88				N/A	Y	12272
> 
>There are no active volume tasks
>
>2.same volume is mounted using NFS and FUSE on client 6.4
>[root@rhs-client22 ~]# mount | grep master1
>10.70.37.128:master1 on /mnt/master1 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
>10.70.37.128:/master1 on /mnt/master1nfs type nfs (rw,addr=10.70.37.128)
>
>
>3. volume is full
>[root@rhs-client22 1]# df -h
>
>10.70.37.128:master1  150G  150G     0 100% /mnt/master1
>10.70.37.128:/master1
>                      150G  150G   64K 100% /mnt/master1nfs
>
>4. from nfs mount ls is not working
>[root@rhs-client22 ~]# ls /mnt/master1nfs
>[root@rhs-client22 ~]# cd /mnt/master1nfs
>[root@rhs-client22 master1nfs]# ls
>[root@rhs-client22 master1nfs]# cd n1
>[root@rhs-client22 n1]# ls
>
>
>FUSE mount shows directories twice
>[root@rhs-client22 n1]# cd /mnt/master1
>[root@rhs-client22 master1]# ls
>1  1  dir1  dir1  f1  f10  f2  f3  f4  f4  f5  f6  f7  f8  f8  f9  n1  n1
>[root@rhs-client22 master1]# cd 1
>[root@rhs-client22 1]# ls
>dir3  dir30  dir31  dir32  dir33  dir34  dir35  dir36  dir37  dir38  dir39  dir4  dir40  dir5  dir6  dir7  dir8  dir9  etc3
>dir3  dir30  dir31  dir32  dir33  dir34  dir35  dir36  dir37  dir38  dir39  dir4  dir40  dir5  dir6  dir7  dir8  dir9  etc3
>
>[root@rhs-client22 1]# cd ../n1
>[root@rhs-client22 n1]# stat *
>  File: `1'
>  Size: 24576     	Blocks: 48         IO Block: 131072 directory
>Device: 16h/22d	Inode: 10770769711530124537  Links: 138
>Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
>Access: 2013-08-20 06:52:51.118999982 -0400
>Modify: 2013-08-20 19:16:31.830999982 -0400
>Change: 2013-08-20 20:48:24.709999982 -0400
>  File: `1'
>  Size: 24576     	Blocks: 48         IO Block: 131072 directory
>Device: 16h/22d	Inode: 10770769711530124537  Links: 138
>Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
>Access: 2013-08-20 06:52:51.118999982 -0400
>Modify: 2013-08-20 19:16:31.830999982 -0400
>Change: 2013-08-20 20:48:24.709999982 -0400
>[root@rhs-client22 n1]# ls
>1  1
>[root@rhs-client22 n1]# ls -il
>total 48
>10770769711530124537 drwxr-xr-x. 138 root root 24576 Aug 20 19:16 1
>10770769711530124537 drwxr-xr-x. 138 root root 24576 Aug 20 19:16 1
>
>
>
>--> verified on bricks, single entity is present
>DVM1
>[root@DVM1 ~]# 
>[root@DVM1 ~]# df -h /rhs/brick1
>Filesystem            Size  Used Avail Use% Mounted on
>/dev/mapper/RHS_vg1-RHS_lv1
>                       50G   50G   20K 100% /rhs/brick1
>[root@DVM1 ~]# ls /rhs/brick1
>1  dir1  f1  f10  f2  f3  f4  f5  f6  f7  f8  f9  n1
>
>
>DVM2
>[root@DVM2 ~]# df -h /rhs/brick1
>Filesystem            Size  Used Avail Use% Mounted on
>/dev/mapper/RHS_vg1-RHS_lv1
>                       50G   50G   20K 100% /rhs/brick1
>[root@DVM2 ~]# ls /rhs/brick1
>1  dir1  f1  f10  f2  f3  f4  f5  f6  f7  f8  f9  n1
>
>DVM4
>[root@DVM4 ~]# df -h /rhs/brick1
>Filesystem            Size  Used Avail Use% Mounted on
>/dev/mapper/RHS_vg1-RHS_lv1
>                       50G   50G   20K 100% /rhs/brick1
>[root@DVM4 ~]# ls /rhs/brick1
>1  dir1  f4  f8  n1
>
>
>DVM5
>[root@DVM5 ~]# df -h /rhs/brick1
>Filesystem            Size  Used Avail Use% Mounted on
>/dev/mapper/RHS_vg1-RHS_lv1
>                       51G   51G   20K 100% /rhs/brick1
>[root@DVM5 ~]# ls /rhs/brick1
>1  dir1  f3  f6  f7  f9  n1
>
>
>DVM6 
>[root@DVM6 ~]# df -h /rhs/brick1
>Filesystem            Size  Used Avail Use% Mounted on
>/dev/mapper/RHS_vg1-RHS_lv1
>                       50G   50G   20K 100% /rhs/brick1
>[root@DVM6 ~]# df -h /rhs/brick5/2/
>Filesystem            Size  Used Avail Use% Mounted on
>/dev/mapper/RHS_vg5-RHS_lv5
>                       50G   50G     0 100% /rhs/brick5
>[root@DVM6 ~]# ls /rhs/brick1
>1  dir1  f4  f8  n1
>[root@DVM6 ~]# ls /rhs/brick5/2/
>1  dir1  f3  f6  f7  f9  n1
>
>
>
>Actual results:
>When volume is full , ls on nfs mount is not showing any data and FUSE mount shows 2 entries for the same entity
>
>Expected results:
>
>
>Additional info:
>ls on nfs mount point and
>
>nfs log shows :-
>[2013-08-21 01:16:06.740835] T [socket.c:741:__socket_nodelay] 0-nfs-server: NODELAY enabled for socket 26
>[2013-08-21 01:16:06.740971] T [socket.c:800:__socket_keepalive] 0-nfs-server: Keep-alive enabled for socket 26, interval 2, idle: 20
>[2013-08-21 01:16:06.741123] T [rpcsvc.c:498:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 744
>[2013-08-21 01:16:06.741201] T [rpcsvc-auth.c:326:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
>[2013-08-21 01:16:06.741229] T [rpcsvc.c:388:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xee1404f8, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1) from rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.741259] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: rhs-client22.lab.eng.blr.redhat., uid: 0, gid: 0
>[2013-08-21 01:16:06.741282] T [rpcsvc.c:214:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - GETATTR
>[2013-08-21 01:16:06.741357] D [nfs3-helpers.c:1629:nfs3_log_common_call] 0-nfs-nfsv3: XID: ee1404f8, GETATTR: args: FH: exportid fa11e206-d039-4606-92fa-29f29a9a8dfa, gfid 00000000-0000-0000-0000-000000000001
>[2013-08-21 01:16:06.741535] T [nfs3.c:839:nfs3_getattr] 0-nfs-nfsv3: FH to Volume: master1
>[2013-08-21 01:16:06.742167] T [nfs3-helpers.c:3729:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution
>[2013-08-21 01:16:06.742250] T [nfs3-helpers.c:3537:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved
>[2013-08-21 01:16:06.742289] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.742317] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.742341] T [nfs-fops.c:508:nfs_fop_stat] 0-nfs: Stat: /
>[2013-08-21 01:16:06.742369] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.742412] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.742479] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.742538] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.742570] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.742650] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x64x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.742698] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.742726] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.742795] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x65x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.742881] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.742953] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.742989] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.743039] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x65x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.743074] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.743177] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.743284] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x66x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.743449] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-0: Returning 0, call_child: 0, last_index: -1
>[2013-08-21 01:16:06.743506] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.743624] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.743821] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.743899] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.744017] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.744178] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x71x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.744260] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x64x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.744453] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x66x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.744551] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x65x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.744675] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x65x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.744808] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.744904] D [nfs3-helpers.c:3391:nfs3_log_common_res] 0-nfs-nfsv3: XID: ee1404f8, GETATTR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
>[2013-08-21 01:16:06.744980] T [rpcsvc.c:1081:rpcsvc_submit_generic] 0-rpc-service: Tx message: 88
>[2013-08-21 01:16:06.745012] T [rpcsvc.c:707:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 112, payload: 88, rpc hdr: 24
>[2013-08-21 01:16:06.745140] T [rpcsvc.c:1118:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x3994289400x, Program: NFS3, ProgVers: 3, Proc: 1) to rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.745263] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x71x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.745679] T [rpcsvc.c:498:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 744
>[2013-08-21 01:16:06.745758] T [rpcsvc-auth.c:326:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
>[2013-08-21 01:16:06.745814] T [rpcsvc.c:388:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xef1404f8, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4) from rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.745878] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: rhs-client22.lab.eng.blr.redhat., uid: 0, gid: 0
>[2013-08-21 01:16:06.745943] T [rpcsvc.c:214:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - ACCESS
>[2013-08-21 01:16:06.746017] D [nfs3-helpers.c:1629:nfs3_log_common_call] 0-nfs-nfsv3: XID: ef1404f8, ACCESS: args: FH: exportid fa11e206-d039-4606-92fa-29f29a9a8dfa, gfid 00000000-0000-0000-0000-000000000001
>[2013-08-21 01:16:06.746046] T [nfs3.c:1577:nfs3_access] 0-nfs-nfsv3: FH to Volume: master1
>[2013-08-21 01:16:06.746076] T [nfs3-helpers.c:3729:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution
>[2013-08-21 01:16:06.746166] T [nfs3-helpers.c:3537:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved
>[2013-08-21 01:16:06.746196] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.746244] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.746268] T [nfs-fops.c:461:nfs_fop_access] 0-nfs: Access: /
>[2013-08-21 01:16:06.746318] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.746342] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.746416] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.746492] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.746556] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68
>[2013-08-21 01:16:06.746654] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x66x Program: GlusterFS 3.3, ProgVers: 330, Proc: 22) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.747422] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x66x Program: GlusterFS 3.3, ProgVers: 330, Proc: 22) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.747680] D [nfs3-helpers.c:3391:nfs3_log_common_res] 0-nfs-nfsv3: XID: ef1404f8, ACCESS: NFS: 0(Call completed successfully.), POSIX: 7(Argument list too long)
>[2013-08-21 01:16:06.747722] T [rpcsvc.c:1081:rpcsvc_submit_generic] 0-rpc-service: Tx message: 12
>[2013-08-21 01:16:06.747748] T [rpcsvc.c:707:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 36, payload: 12, rpc hdr: 24
>[2013-08-21 01:16:06.747816] T [rpcsvc.c:1118:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x4011066616x, Program: NFS3, ProgVers: 3, Proc: 4) to rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.748216] T [rpcsvc.c:498:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 744
>[2013-08-21 01:16:06.748305] T [rpcsvc-auth.c:326:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
>[2013-08-21 01:16:06.748334] T [rpcsvc.c:388:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xf01404f8, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1) from rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.748360] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: rhs-client22.lab.eng.blr.redhat., uid: 0, gid: 0
>[2013-08-21 01:16:06.748402] T [rpcsvc.c:214:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - GETATTR
>[2013-08-21 01:16:06.748435] D [nfs3-helpers.c:1629:nfs3_log_common_call] 0-nfs-nfsv3: XID: f01404f8, GETATTR: args: FH: exportid fa11e206-d039-4606-92fa-29f29a9a8dfa, gfid 00000000-0000-0000-0000-000000000001
>[2013-08-21 01:16:06.748492] T [nfs3.c:839:nfs3_getattr] 0-nfs-nfsv3: FH to Volume: master1
>[2013-08-21 01:16:06.748525] T [nfs3-helpers.c:3729:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution
>[2013-08-21 01:16:06.748579] T [nfs3-helpers.c:3537:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved
>[2013-08-21 01:16:06.748609] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.748657] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.748681] T [nfs-fops.c:508:nfs_fop_stat] 0-nfs: Stat: /
>[2013-08-21 01:16:06.748729] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.748752] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.748817] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.748890] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.748952] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.749025] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x67x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.749145] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.749217] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.749276] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.749339] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x67x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.749422] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-0: Returning 0, call_child: 0, last_index: -1
>[2013-08-21 01:16:06.749496] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.749555] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.749676] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x72x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.749893] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x67x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.750001] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x67x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.750062] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x72x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.750170] D [nfs3-helpers.c:3391:nfs3_log_common_res] 0-nfs-nfsv3: XID: f01404f8, GETATTR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
>[2013-08-21 01:16:06.750208] T [rpcsvc.c:1081:rpcsvc_submit_generic] 0-rpc-service: Tx message: 88
>[2013-08-21 01:16:06.750233] T [rpcsvc.c:707:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 112, payload: 88, rpc hdr: 24
>[2013-08-21 01:16:06.750293] T [rpcsvc.c:1118:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x4027843832x, Program: NFS3, ProgVers: 3, Proc: 1) to rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.750738] T [rpcsvc.c:498:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 744
>[2013-08-21 01:16:06.750870] T [rpcsvc-auth.c:326:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
>[2013-08-21 01:16:06.750937] T [rpcsvc.c:388:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xf11404f8, Ver: 2, Program: 100003, ProgVers: 3, Proc: 17) from rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.750995] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: rhs-client22.lab.eng.blr.redhat., uid: 0, gid: 0
>[2013-08-21 01:16:06.751023] T [rpcsvc.c:214:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - READDIRPLUS
>[2013-08-21 01:16:06.751057] D [nfs3-helpers.c:3525:nfs3_log_readdir_call] 0-nfs-nfsv3: XID: f11404f8, READDIRPLUS: args: FH: exportid fa11e206-d039-4606-92fa-29f29a9a8dfa, gfid 00000000-0000-0000-0000-000000000001, dircount: 4096, maxcount: 32768
>[2013-08-21 01:16:06.751117] T [nfs3.c:4327:nfs3_readdir] 0-nfs-nfsv3: FH to Volume: master1
>[2013-08-21 01:16:06.751182] T [nfs3-helpers.c:3729:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution
>[2013-08-21 01:16:06.751210] T [nfs3-helpers.c:3537:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved
>[2013-08-21 01:16:06.751240] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.751260] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.751283] T [nfs-fops.c:600:nfs_fop_opendir] 0-nfs: Opendir: /
>[2013-08-21 01:16:06.751337] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.751361] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.751466] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.751533] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.751609] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x73x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.751657] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-1: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.751684] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.751773] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.751881] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752002] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.752074] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x49x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.752209] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-2: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752273] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.752322] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x52x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.752406] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752467] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.752516] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x53x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.752589] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752621] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.752666] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x68x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.752819] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-4: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752891] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.753057] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.753193] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-4: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.753296] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.753369] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x52x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.753494] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.753560] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.753660] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x68x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.753738] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.753859] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.753888] T [nfs-fops.c:690:nfs_fop_readdirp] 0-nfs: readdir
>[2013-08-21 01:16:06.754004] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.754065] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.754237] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-0: Returning 0, call_child: 0, last_index: -1
>[2013-08-21 01:16:06.754384] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.754457] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68
>[2013-08-21 01:16:06.754532] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x74x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.754635] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x73x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.754826] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-1: received rpc message (RPC XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.754926] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-2: received rpc message (RPC XID: 0x52x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.755013] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x68x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.755181] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-4: received rpc message (RPC XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.755283] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x68x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.755344] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-0: sending releasedir on fd
>[2013-08-21 01:16:06.755425] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.755488] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.755561] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x75x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.755664] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-1: sending releasedir on fd
>[2013-08-21 01:16:06.755738] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.755798] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.755927] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.755992] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-2: sending releasedir on fd
>[2013-08-21 01:16:06.756023] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.756078] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.756211] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x54x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.756290] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-3: sending releasedir on fd
>[2013-08-21 01:16:06.756354] T [rpc-clnt.c:1307:rpc_clnt_reco[2013-08-21 01:16:06.740835] T [socket.c:741:__socket_nodelay] 0-nfs-server: NODELAY enabled for socket 26
>[2013-08-21 01:16:06.740971] T [socket.c:800:__socket_keepalive] 0-nfs-server: Keep-alive enabled for socket 26, interval 2, idle: 20
>[2013-08-21 01:16:06.741123] T [rpcsvc.c:498:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 744
>[2013-08-21 01:16:06.741201] T [rpcsvc-auth.c:326:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
>[2013-08-21 01:16:06.741229] T [rpcsvc.c:388:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xee1404f8, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1) from rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.741259] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: rhs-client22.lab.eng.blr.redhat., uid: 0, gid: 0
>[2013-08-21 01:16:06.741282] T [rpcsvc.c:214:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - GETATTR
>[2013-08-21 01:16:06.741357] D [nfs3-helpers.c:1629:nfs3_log_common_call] 0-nfs-nfsv3: XID: ee1404f8, GETATTR: args: FH: exportid fa11e206-d039-4606-92fa-29f29a9a8dfa, gfid 00000000-0000-0000-0000-000000000001
>[2013-08-21 01:16:06.741535] T [nfs3.c:839:nfs3_getattr] 0-nfs-nfsv3: FH to Volume: master1
>[2013-08-21 01:16:06.742167] T [nfs3-helpers.c:3729:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution
>[2013-08-21 01:16:06.742250] T [nfs3-helpers.c:3537:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved
>[2013-08-21 01:16:06.742289] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.742317] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.742341] T [nfs-fops.c:508:nfs_fop_stat] 0-nfs: Stat: /
>[2013-08-21 01:16:06.742369] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.742412] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.742479] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.742538] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.742570] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.742650] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x64x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.742698] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.742726] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.742795] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x65x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.742881] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.742953] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.742989] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.743039] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x65x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.743074] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.743177] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.743284] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x66x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.743449] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-0: Returning 0, call_child: 0, last_index: -1
>[2013-08-21 01:16:06.743506] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.743624] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.743821] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.743899] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.744017] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.744178] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x71x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.744260] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x64x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.744453] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x66x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.744551] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x65x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.744675] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x65x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.744808] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.744904] D [nfs3-helpers.c:3391:nfs3_log_common_res] 0-nfs-nfsv3: XID: ee1404f8, GETATTR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
>[2013-08-21 01:16:06.744980] T [rpcsvc.c:1081:rpcsvc_submit_generic] 0-rpc-service: Tx message: 88
>[2013-08-21 01:16:06.745012] T [rpcsvc.c:707:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 112, payload: 88, rpc hdr: 24
>[2013-08-21 01:16:06.745140] T [rpcsvc.c:1118:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x3994289400x, Program: NFS3, ProgVers: 3, Proc: 1) to rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.745263] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x71x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.745679] T [rpcsvc.c:498:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 744
>[2013-08-21 01:16:06.745758] T [rpcsvc-auth.c:326:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
>[2013-08-21 01:16:06.745814] T [rpcsvc.c:388:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xef1404f8, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4) from rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.745878] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: rhs-client22.lab.eng.blr.redhat., uid: 0, gid: 0
>[2013-08-21 01:16:06.745943] T [rpcsvc.c:214:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - ACCESS
>[2013-08-21 01:16:06.746017] D [nfs3-helpers.c:1629:nfs3_log_common_call] 0-nfs-nfsv3: XID: ef1404f8, ACCESS: args: FH: exportid fa11e206-d039-4606-92fa-29f29a9a8dfa, gfid 00000000-0000-0000-0000-000000000001
>[2013-08-21 01:16:06.746046] T [nfs3.c:1577:nfs3_access] 0-nfs-nfsv3: FH to Volume: master1
>[2013-08-21 01:16:06.746076] T [nfs3-helpers.c:3729:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution
>[2013-08-21 01:16:06.746166] T [nfs3-helpers.c:3537:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved
>[2013-08-21 01:16:06.746196] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.746244] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.746268] T [nfs-fops.c:461:nfs_fop_access] 0-nfs: Access: /
>[2013-08-21 01:16:06.746318] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.746342] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.746416] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.746492] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.746556] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68
>[2013-08-21 01:16:06.746654] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x66x Program: GlusterFS 3.3, ProgVers: 330, Proc: 22) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.747422] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x66x Program: GlusterFS 3.3, ProgVers: 330, Proc: 22) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.747680] D [nfs3-helpers.c:3391:nfs3_log_common_res] 0-nfs-nfsv3: XID: ef1404f8, ACCESS: NFS: 0(Call completed successfully.), POSIX: 7(Argument list too long)
>[2013-08-21 01:16:06.747722] T [rpcsvc.c:1081:rpcsvc_submit_generic] 0-rpc-service: Tx message: 12
>[2013-08-21 01:16:06.747748] T [rpcsvc.c:707:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 36, payload: 12, rpc hdr: 24
>[2013-08-21 01:16:06.747816] T [rpcsvc.c:1118:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x4011066616x, Program: NFS3, ProgVers: 3, Proc: 4) to rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.748216] T [rpcsvc.c:498:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 744
>[2013-08-21 01:16:06.748305] T [rpcsvc-auth.c:326:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
>[2013-08-21 01:16:06.748334] T [rpcsvc.c:388:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xf01404f8, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1) from rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.748360] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: rhs-client22.lab.eng.blr.redhat., uid: 0, gid: 0
>[2013-08-21 01:16:06.748402] T [rpcsvc.c:214:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - GETATTR
>[2013-08-21 01:16:06.748435] D [nfs3-helpers.c:1629:nfs3_log_common_call] 0-nfs-nfsv3: XID: f01404f8, GETATTR: args: FH: exportid fa11e206-d039-4606-92fa-29f29a9a8dfa, gfid 00000000-0000-0000-0000-000000000001
>[2013-08-21 01:16:06.748492] T [nfs3.c:839:nfs3_getattr] 0-nfs-nfsv3: FH to Volume: master1
>[2013-08-21 01:16:06.748525] T [nfs3-helpers.c:3729:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution
>[2013-08-21 01:16:06.748579] T [nfs3-helpers.c:3537:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved
>[2013-08-21 01:16:06.748609] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.748657] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.748681] T [nfs-fops.c:508:nfs_fop_stat] 0-nfs: Stat: /
>[2013-08-21 01:16:06.748729] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.748752] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.748817] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.748890] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.748952] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.749025] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x67x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.749145] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.749217] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.749276] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.749339] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x67x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.749422] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-0: Returning 0, call_child: 0, last_index: -1
>[2013-08-21 01:16:06.749496] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.749555] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.749676] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x72x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.749893] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x67x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.750001] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x67x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.750062] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x72x Program: GlusterFS 3.3, ProgVers: 330, Proc: 1) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.750170] D [nfs3-helpers.c:3391:nfs3_log_common_res] 0-nfs-nfsv3: XID: f01404f8, GETATTR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
>[2013-08-21 01:16:06.750208] T [rpcsvc.c:1081:rpcsvc_submit_generic] 0-rpc-service: Tx message: 88
>[2013-08-21 01:16:06.750233] T [rpcsvc.c:707:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 112, payload: 88, rpc hdr: 24
>[2013-08-21 01:16:06.750293] T [rpcsvc.c:1118:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x4027843832x, Program: NFS3, ProgVers: 3, Proc: 1) to rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.750738] T [rpcsvc.c:498:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 744
>[2013-08-21 01:16:06.750870] T [rpcsvc-auth.c:326:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
>[2013-08-21 01:16:06.750937] T [rpcsvc.c:388:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xf11404f8, Ver: 2, Program: 100003, ProgVers: 3, Proc: 17) from rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.750995] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: rhs-client22.lab.eng.blr.redhat., uid: 0, gid: 0
>[2013-08-21 01:16:06.751023] T [rpcsvc.c:214:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - READDIRPLUS
>[2013-08-21 01:16:06.751057] D [nfs3-helpers.c:3525:nfs3_log_readdir_call] 0-nfs-nfsv3: XID: f11404f8, READDIRPLUS: args: FH: exportid fa11e206-d039-4606-92fa-29f29a9a8dfa, gfid 00000000-0000-0000-0000-000000000001, dircount: 4096, maxcount: 32768
>[2013-08-21 01:16:06.751117] T [nfs3.c:4327:nfs3_readdir] 0-nfs-nfsv3: FH to Volume: master1
>[2013-08-21 01:16:06.751182] T [nfs3-helpers.c:3729:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution
>[2013-08-21 01:16:06.751210] T [nfs3-helpers.c:3537:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved
>[2013-08-21 01:16:06.751240] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.751260] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.751283] T [nfs-fops.c:600:nfs_fop_opendir] 0-nfs: Opendir: /
>[2013-08-21 01:16:06.751337] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.751361] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.751466] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.751533] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.751609] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x73x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.751657] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-1: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.751684] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.751773] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.751881] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752002] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.752074] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x49x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.752209] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-2: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752273] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.752322] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x52x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.752406] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752467] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.752516] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x53x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.752589] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752621] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.752666] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x68x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.752819] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-4: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.752891] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.753057] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.753193] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-4: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.753296] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-08-21 01:16:06.753369] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x52x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.753494] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.753560] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 88, payload: 20, rpc hdr: 68
>[2013-08-21 01:16:06.753660] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x68x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.753738] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.753859] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.753888] T [nfs-fops.c:690:nfs_fop_readdirp] 0-nfs: readdir
>[2013-08-21 01:16:06.754004] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.754065] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.754237] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-0: Returning 0, call_child: 0, last_index: -1
>[2013-08-21 01:16:06.754384] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.754457] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68
>[2013-08-21 01:16:06.754532] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x74x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.754635] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x73x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.754826] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-1: received rpc message (RPC XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.754926] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-2: received rpc message (RPC XID: 0x52x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.755013] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x68x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.755181] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-4: received rpc message (RPC XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.755283] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x68x Program: GlusterFS 3.3, ProgVers: 330, Proc: 20) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.755344] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-0: sending releasedir on fd
>[2013-08-21 01:16:06.755425] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.755488] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.755561] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x75x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.755664] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-1: sending releasedir on fd
>[2013-08-21 01:16:06.755738] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.755798] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.755927] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.755992] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-2: sending releasedir on fd
>[2013-08-21 01:16:06.756023] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.756078] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.756211] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x54x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.756290] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-3: sending releasedir on fd
>[2013-08-21 01:16:06.756354] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.756425] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.756493] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x69x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.756559] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-4: sending releasedir on fd
>[2013-08-21 01:16:06.756591] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-4: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.756651] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.756772] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x53x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.756993] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-5: sending releasedir on fd
>[2013-08-21 01:16:06.757033] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.757118] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.757193] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x69x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.757323] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x74x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.757407] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 0-master1-client-0: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.757456] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.757518] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.757626] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68
>[2013-08-21 01:16:06.757712] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.757808] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-1: received rpc message (RPC XID: 0x49x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.757859] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-2: received rpc message (RPC XID: 0x53x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.757935] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x69x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.758008] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-4: received rpc message (RPC XID: 0x52x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.758159] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x75x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.758244] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-1: received rpc message (RPC XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.758287] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-2: received rpc message (RPC XID: 0x54x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.758324] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-4: received rpc message (RPC XID: 0x53x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.758404] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x69x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.758494] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.758560] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 0-master1-client-3: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.758655] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.758736] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.758913] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68
>[2013-08-21 01:16:06.759015] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.760790] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.760951] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.761122] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.761197] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68
>[2013-08-21 01:16:06.761276] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x71x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.762019] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x71x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.762152] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.762185] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.762205] T [nfs-fops.c:555:nfs_fop_fstat] 0-nfs: FStat
>[2013-08-21 01:16:06.762225] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.762244] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.762317] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.762421] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.762507] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.762616] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x71x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.762777] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.762851] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.762960] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.763068] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x72x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.763197] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-0: Returning 0, call_child: 0, last_index: -1
>[2013-08-21 01:16:06.763293] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.763357] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.763423] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x76x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.763617] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x71x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.763719] W [client-rpc-fops.c:1479:client3_3_fstat_cbk] 0-master1-client-3: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.763790] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-2: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.763851] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.764021] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x55x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.764149] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x72x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.764250] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x76x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.764286] W [client-rpc-fops.c:1479:client3_3_fstat_cbk] 0-master1-client-0: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.764349] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-1: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.764379] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.764460] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.764913] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-2: received rpc message (RPC XID: 0x55x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.765011] W [client-rpc-fops.c:1479:client3_3_fstat_cbk] 0-master1-client-2: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.765048] D [dht-inode-read.c:236:dht_attr_cbk] 0-master1-dht: subvolume master1-replicate-1 returned -1 (Operation not permitted)
>[2013-08-21 01:16:06.765198] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-1: received rpc message (RPC XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.765275] W [client-rpc-fops.c:1479:client3_3_fstat_cbk] 0-master1-client-1: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.765335] D [dht-inode-read.c:236:dht_attr_cbk] 0-master1-dht: subvolume master1-replicate-0 returned -1 (Operation not permitted)
>[2013-08-21 01:16:06.765393] T [nfs3.c:4087:nfs3svc_readdir_fstat_cbk] 0-nfs-nfsv3: Reached end-of-directory
>[2013-08-21 01:16:06.765432] D [nfs3-helpers.c:3491:nfs3_log_readdirp_res] 0-nfs-nfsv3: XID: f11404f8, READDIRPLUS: NFS: 0(Call completed successfully.), POSIX: 1(Operation not permitted), dircount: 4096, maxcount: 32768, cverf: 16116808, is_eof: 1
>[2013-08-21 01:16:06.765503] T [rpcsvc.c:1081:rpcsvc_submit_generic] 0-rpc-service: Tx message: 108
>[2013-08-21 01:16:06.765565] T [rpcsvc.c:707:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 132, payload: 108, rpc hdr: 24
>[2013-08-21 01:16:06.765675] T [rpcsvc.c:1118:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x4044621048x, Program: NFS3, ProgVers: 3, Proc: 17) to rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.765745] T [nfs3.c:446:nfs3_call_state_wipe] 0-nfs-nfsv3: fd 0xf5ec48 ref: 3
>rd] 0-master1-client-3: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.756425] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.756493] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x69x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.756559] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-4: sending releasedir on fd
>[2013-08-21 01:16:06.756591] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-4: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.756651] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.756772] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x53x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.756993] T [client-rpc-fops.c:2760:client_fdctx_destroy] 0-master1-client-5: sending releasedir on fd
>[2013-08-21 01:16:06.757033] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.757118] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-08-21 01:16:06.757193] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x69x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.757323] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x74x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.757407] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 0-master1-client-0: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.757456] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.757518] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.757626] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68
>[2013-08-21 01:16:06.757712] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.757808] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-1: received rpc message (RPC XID: 0x49x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.757859] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-2: received rpc message (RPC XID: 0x53x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.757935] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x69x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.758008] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-4: received rpc message (RPC XID: 0x52x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.758159] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x75x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.758244] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-1: received rpc message (RPC XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.758287] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-2: received rpc message (RPC XID: 0x54x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.758324] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-4: received rpc message (RPC XID: 0x53x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-4)
>[2013-08-21 01:16:06.758404] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x69x Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.758494] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.758560] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 0-master1-client-3: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.758655] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.758736] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.758913] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68
>[2013-08-21 01:16:06.759015] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.760790] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x70x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.760951] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.761122] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.761197] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68
>[2013-08-21 01:16:06.761276] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x71x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.762019] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x71x Program: GlusterFS 3.3, ProgVers: 330, Proc: 40) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.762152] T [nfs.c:454:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.762185] T [nfs.c:462:nfs_user_create] 0-nfs: gid: 0
>[2013-08-21 01:16:06.762205] T [nfs-fops.c:555:nfs_fop_fstat] 0-nfs: FStat
>[2013-08-21 01:16:06.762225] T [nfs-fops.c:203:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1
>[2013-08-21 01:16:06.762244] T [nfs-fops.c:205:nfs_create_frame] 0-nfs: gid: 0
>[2013-08-21 01:16:06.762317] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-1: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.762421] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-3: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.762507] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.762616] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x71x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.762777] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-2: Returning 0, call_child: 1, last_index: -1
>[2013-08-21 01:16:06.762851] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-5: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.762960] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.763068] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x72x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.763197] D [afr-common.c:745:afr_get_call_child] 0-master1-replicate-0: Returning 0, call_child: 0, last_index: -1
>[2013-08-21 01:16:06.763293] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.763357] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.763423] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x76x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.763617] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-3: received rpc message (RPC XID: 0x71x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-3)
>[2013-08-21 01:16:06.763719] W [client-rpc-fops.c:1479:client3_3_fstat_cbk] 0-master1-client-3: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.763790] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-2: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.763851] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.764021] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x55x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.764149] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-5: received rpc message (RPC XID: 0x72x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-5)
>[2013-08-21 01:16:06.764250] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-0: received rpc message (RPC XID: 0x76x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-0)
>[2013-08-21 01:16:06.764286] W [client-rpc-fops.c:1479:client3_3_fstat_cbk] 0-master1-client-0: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.764349] T [rpc-clnt.c:1307:rpc_clnt_record] 0-master1-client-1: Auth Info: pid: 1, uid: 0, gid: 0, owner: 
>[2013-08-21 01:16:06.764379] T [rpc-clnt.c:1187:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-08-21 01:16:06.764460] T [rpc-clnt.c:1504:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.764913] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-2: received rpc message (RPC XID: 0x55x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-2)
>[2013-08-21 01:16:06.765011] W [client-rpc-fops.c:1479:client3_3_fstat_cbk] 0-master1-client-2: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.765048] D [dht-inode-read.c:236:dht_attr_cbk] 0-master1-dht: subvolume master1-replicate-1 returned -1 (Operation not permitted)
>[2013-08-21 01:16:06.765198] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-master1-client-1: received rpc message (RPC XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (master1-client-1)
>[2013-08-21 01:16:06.765275] W [client-rpc-fops.c:1479:client3_3_fstat_cbk] 0-master1-client-1: remote operation failed: Operation not permitted
>[2013-08-21 01:16:06.765335] D [dht-inode-read.c:236:dht_attr_cbk] 0-master1-dht: subvolume master1-replicate-0 returned -1 (Operation not permitted)
>[2013-08-21 01:16:06.765393] T [nfs3.c:4087:nfs3svc_readdir_fstat_cbk] 0-nfs-nfsv3: Reached end-of-directory
>[2013-08-21 01:16:06.765432] D [nfs3-helpers.c:3491:nfs3_log_readdirp_res] 0-nfs-nfsv3: XID: f11404f8, READDIRPLUS: NFS: 0(Call completed successfully.), POSIX: 1(Operation not permitted), dircount: 4096, maxcount: 32768, cverf: 16116808, is_eof: 1
>[2013-08-21 01:16:06.765503] T [rpcsvc.c:1081:rpcsvc_submit_generic] 0-rpc-service: Tx message: 108
>[2013-08-21 01:16:06.765565] T [rpcsvc.c:707:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 132, payload: 108, rpc hdr: 24
>[2013-08-21 01:16:06.765675] T [rpcsvc.c:1118:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x4044621048x, Program: NFS3, ProgVers: 3, Proc: 17) to rpc-transport (socket.nfs-server)
>[2013-08-21 01:16:06.765745] T [nfs3.c:446:nfs3_call_state_wipe] 0-nfs-nfsv3: fd 0xf5ec48 ref: 3

Actions: View

Attachments on bug 999399: 915750