Bug 763116 (GLUSTER-1384)

Summary: gnfs crashed on inode_table_prune during hard fh resolution
Product: [Community] GlusterFS Reporter: Lakshmipathi G <lakshmipathi>
Component: nfsAssignee: Shehjar Tikoo <shehjart>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: low    
Version: 3.1-alphaCC: gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: nfs
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lakshmipathi G 2010-08-17 05:33:28 UTC 
while running mixed tests in 4x3 dht-afr with 3.1.0qa5-15-gc87c868 gnfs crashed.
log file:
-----------------------------------------------------
[2010-08-16 13:50:47.666069] T [rpcsvc.c:1085:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Regular RPC header, remaining: 128
[2010-08-16 13:50:47.666093] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x70ff00, readsize: 4, dataread:4
[2010-08-16 13:50:47.666108] T [rpcsvc.c:1078:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Received fragment size: 132
[2010-08-16 13:50:47.666123] T [rpcsvc.c:1085:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Regular RPC header, remaining: 132
[2010-08-16 13:50:47.666142] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x70ff00, readsize: 132, dataread:132
[2010-08-16 13:50:47.666157] T [rpcsvc.c:2360:nfs_rpcsvc_record_update_state] nfsrpc: Regular frag
[2010-08-16 13:50:47.666184] T [rpcsvc.c:1176:nfs_rpcsvc_record_read_complete_frag] nfsrpc: Fragment remaining: 0
[2010-08-16 13:50:47.666201] T [rpcsvc.c:2382:nfs_rpcsvc_record_update_state] nfsrpc: Full Record Received.
[2010-08-16 13:50:47.666217] T [rpcsvc-auth.c:247:nfs_rpcsvc_auth_request_init] nfsrpc: Auth handler: AUTH_NULL
[2010-08-16 13:50:47.666232] D [rpcsvc.c:1842:nfs_rpcsvc_request_create] nfsrpc: RPC XID: 6f88ef45, Ver: 2, Program: 100003, ProgVers: 3, Proc: 2
[2010-08-16 13:50:47.666248] D [rpcsvc.c:1285:nfs_rpcsvc_program_actor] nfsrpc: Actor found: NFS3 - SETATTR
[2010-08-16 13:50:47.666266] D [nfs3-helpers.c:2227:nfs3_log_common_call] nfs-nfsv3: XID: 6f88ef45, SETATTR: args: FH: hashcount 11, xlid 0, gen 5506008053243323329, ino 96345026
[2010-08-16 13:50:47.666282] T [nfs3.c:770:nfs3_setattr] nfs-nfsv3: FH to Volume: statprefetch
[2010-08-16 13:50:47.666297] T [nfs3.c:781:nfs3_setattr] nfs-nfsv3: Guard check not required
[2010-08-16 13:50:47.666311] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-16 13:50:47.666352] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-16 13:50:47.666385] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-16 13:50:47.666405] T [nfs-fops.c:649:nfs_fop_setattr] nfs: Setattr: /nfsalpha201/ip-10-244-167-207/test7/linux-2.6.35/arch/arm/mach-bcmring/include/mach/csp/mm_addr.h
[2010-08-16 13:50:47.666428] T [io-cache.c:133:ioc_inode_flush] iocache: locked inode(0x2aaafc68b0d0)
[2010-08-16 13:50:47.666444] T [io-cache.c:137:ioc_inode_flush] iocache: unlocked inode(0x2aaafc68b0d0)
[2010-08-16 13:50:47.666463] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.666494] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.666513] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 352, payload: 224, rpc hdr: 128
[2010-08-16 13:50:47.666548] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.666566] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 352, payload: 224, rpc hdr: 128
[2010-08-16 13:50:47.666589] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.666605] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 352, payload: 224, rpc hdr: 128
[2010-08-16 13:50:47.666633] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x675380, readsize: 128, dataread:128
[2010-08-16 13:50:47.666649] T [rpcsvc.c:2360:nfs_rpcsvc_record_update_state] nfsrpc: Regular frag
[2010-08-16 13:50:47.666664] T [rpcsvc.c:1176:nfs_rpcsvc_record_read_complete_frag] nfsrpc: Fragment remaining: 0
[2010-08-16 13:50:47.666679] T [rpcsvc.c:2382:nfs_rpcsvc_record_update_state] nfsrpc: Full Record Received.
[2010-08-16 13:50:47.666695] T [rpcsvc-auth.c:247:nfs_rpcsvc_auth_request_init] nfsrpc: Auth handler: AUTH_NULL
[2010-08-16 13:50:47.666710] D [rpcsvc.c:1842:nfs_rpcsvc_request_create] nfsrpc: RPC XID: 8eb1f7dd, Ver: 2, Program: 100003, ProgVers: 3, Proc: 2
[2010-08-16 13:50:47.666727] D [rpcsvc.c:1285:nfs_rpcsvc_program_actor] nfsrpc: Actor found: NFS3 - SETATTR
[2010-08-16 13:50:47.666746] D [nfs3-helpers.c:2227:nfs3_log_common_call] nfs-nfsv3: XID: 8eb1f7dd, SETATTR: args: FH: hashcount 9, xlid 0, gen 5506007464832804124, ino 40913460
[2010-08-16 13:50:47.666788] T [nfs3.c:770:nfs3_setattr] nfs-nfsv3: FH to Volume: statprefetch
[2010-08-16 13:50:47.666813] T [nfs3.c:781:nfs3_setattr] nfs-nfsv3: Guard check not required
[2010-08-16 13:50:47.666827] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-16 13:50:47.666844] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-16 13:50:47.666867] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-16 13:50:47.666882] T [nfs-fops.c:649:nfs_fop_setattr] nfs: Setattr: /nfsalpha101/ip-10-204-110-127/test6/linux-2.6.35/arch/arm/include/asm/swab.h
[2010-08-16 13:50:47.666913] T [io-cache.c:133:ioc_inode_flush] iocache: locked inode(0x2aab005521e0)
[2010-08-16 13:50:47.666930] T [io-cache.c:137:ioc_inode_flush] iocache: unlocked inode(0x2aab005521e0)
[2010-08-16 13:50:47.666948] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.666978] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.666996] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 332, payload: 204, rpc hdr: 128
[2010-08-16 13:50:47.667021] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.667038] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 332, payload: 204, rpc hdr: 128
[2010-08-16 13:50:47.667083] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x675380, readsize: 4, dataread:4
[2010-08-16 13:50:47.667101] T [rpcsvc.c:1078:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Received fragment size: 100
[2010-08-16 13:50:47.667116] T [rpcsvc.c:1085:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Regular RPC header, remaining: 100
[2010-08-16 13:50:47.667134] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x70ff00, readsize: 4, dataread:4
[2010-08-16 13:50:47.667150] T [rpcsvc.c:1078:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Received fragment size: 124
[2010-08-16 13:50:47.667165] T [rpcsvc.c:1085:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Regular RPC header, remaining: 124
[2010-08-16 13:50:47.667184] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x70ff00, readsize: 124, dataread:124
[2010-08-16 13:50:47.667199] T [rpcsvc.c:2360:nfs_rpcsvc_record_update_state] nfsrpc: Regular frag
[2010-08-16 13:50:47.667214] T [rpcsvc.c:1176:nfs_rpcsvc_record_read_complete_frag] nfsrpc: Fragment remaining: 0
[2010-08-16 13:50:47.667229] T [rpcsvc.c:2382:nfs_rpcsvc_record_update_state] nfsrpc: Full Record Received.
[2010-08-16 13:50:47.667245] T [rpcsvc-auth.c:247:nfs_rpcsvc_auth_request_init] nfsrpc: Auth handler: AUTH_NULL
[2010-08-16 13:50:47.667259] D [rpcsvc.c:1842:nfs_rpcsvc_request_create] nfsrpc: RPC XID: 7088ef45, Ver: 2, Program: 100003, ProgVers: 3, Proc: 2
[2010-08-16 13:50:47.667275] D [rpcsvc.c:1285:nfs_rpcsvc_program_actor] nfsrpc: Actor found: NFS3 - SETATTR
[2010-08-16 13:50:47.667293] D [nfs3-helpers.c:2227:nfs3_log_common_call] nfs-nfsv3: XID: 7088ef45, SETATTR: args: FH: hashcount 7, xlid 0, gen 5506008053243323328, ino 302579786
[2010-08-16 13:50:47.667309] T [nfs3.c:770:nfs3_setattr] nfs-nfsv3: FH to Volume: statprefetch
[2010-08-16 13:50:47.667326] T [nfs3.c:781:nfs3_setattr] nfs-nfsv3: Guard check not required
[2010-08-16 13:50:47.667342] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-16 13:50:47.667358] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-16 13:50:47.667380] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-16 13:50:47.667486] T [nfs-fops.c:649:nfs_fop_setattr] nfs: Setattr: /nfsalpha202/ip-10-244-167-207/test8/linux-2.6.35/Documentation/zh_CN/.sparse.txt.nqS1z8
[2010-08-16 13:50:47.667515] T [io-cache.c:133:ioc_inode_flush] iocache: locked inode(0x2aab0052d6a0)
[2010-08-16 13:50:47.667530] T [io-cache.c:137:ioc_inode_flush] iocache: unlocked inode(0x2aab0052d6a0)
[2010-08-16 13:50:47.667558] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.667575] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 340, payload: 212, rpc hdr: 128
[2010-08-16 13:50:47.667601] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.667617] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 340, payload: 212, rpc hdr: 128
[2010-08-16 13:50:47.667640] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.667656] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 340, payload: 212, rpc hdr: 128
[2010-08-16 13:50:47.667695] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x675380, readsize: 100, dataread:100
[2010-08-16 13:50:47.667712] T [rpcsvc.c:2360:nfs_rpcsvc_record_update_state] nfsrpc: Regular frag
[2010-08-16 13:50:47.667727] T [rpcsvc.c:1176:nfs_rpcsvc_record_read_complete_frag] nfsrpc: Fragment remaining: 0
[2010-08-16 13:50:47.667742] T [rpcsvc.c:2382:nfs_rpcsvc_record_update_state] nfsrpc: Full Record Received.
[2010-08-16 13:50:47.667758] T [rpcsvc-auth.c:247:nfs_rpcsvc_auth_request_init] nfsrpc: Auth handler: AUTH_NULL
[2010-08-16 13:50:47.667773] D [rpcsvc.c:1842:nfs_rpcsvc_request_create] nfsrpc: RPC XID: 8fb1f7dd, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3
[2010-08-16 13:50:47.667790] D [rpcsvc.c:1285:nfs_rpcsvc_program_actor] nfsrpc: Actor found: NFS3 - LOOKUP
[2010-08-16 13:50:47.667812] D [nfs3-helpers.c:2239:nfs3_log_fh_entry_call] nfs-nfsv3: XID: 8fb1f7dd, LOOKUP: args: FH: hashcount 7, xlid 0, gen 5506007464832714316, ino 656673360, name: pcm027_defconfig
[2010-08-16 13:50:47.667828] T [nfs3.c:1040:nfs3_lookup] nfs-nfsv3: FH to Volume: statprefetch
[2010-08-16 13:50:47.667860] T [nfs3-helpers.c:2940:nfs3_fh_resolve_entry_hard] nfs-nfsv3: FH hard resolution: ino: 656673360, gen: 5506007464832714316, entry: pcm027_defconfig, hashidx: 0
[2010-08-16 13:50:47.667888] T [nfs3-helpers.c:2948:nfs3_fh_resolve_entry_hard] nfs-nfsv3: Entry needs lookup: /nfsalpha101/ip-10-204-110-127/test8/linux-2.6.35/arch/arm/configs/pcm027_defconfig
[2010-08-16 13:50:47.667904] T [nfs-fops.c:279:nfs_fop_lookup] nfs: Lookup: /nfsalpha101/ip-10-204-110-127/test8/linux-2.6.35/arch/arm/configs/pcm027_defconfig
[2010-08-16 13:50:47.667964] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891153952, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.667981] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 692, payload: 564, rpc hdr: 128
[2010-08-16 13:50:47.668022] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891153952, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.668039] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 692, payload: 564, rpc hdr: 128
[2010-08-16 13:50:47.668074] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x675380, readsize: 4, dataread:4
[2010-08-16 13:50:47.668108] T [rpcsvc.c:1078:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Received fragment size: 100
[2010-08-16 13:50:47.668123] T [rpcsvc.c:1085:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Regular RPC header, remaining: 100
[2010-08-16 13:50:47.668140] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x70ff00, readsize: 4, dataread:4
[2010-08-16 13:50:47.668155] T [rpcsvc.c:1078:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Received fragment size: 84
[2010-08-16 13:50:47.668169] T [rpcsvc.c:1085:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Regular RPC header, remaining: 84
[2010-08-16 13:50:47.668187] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x70ff00, readsize: 84, dataread:84
[2010-08-16 13:50:47.668203] T [rpcsvc.c:2360:nfs_rpcsvc_record_update_state] nfsrpc: Regular frag
[2010-08-16 13:50:47.668217] T [rpcsvc.c:1176:nfs_rpcsvc_record_read_complete_frag] nfsrpc: Fragment remaining: 0
[2010-08-16 13:50:47.668231] T [rpcsvc.c:2382:nfs_rpcsvc_record_update_state] nfsrpc: Full Record Received.
[2010-08-16 13:50:47.668252] T [rpcsvc-auth.c:247:nfs_rpcsvc_auth_request_init] nfsrpc: Auth handler: AUTH_NULL
[2010-08-16 13:50:47.668276] D [rpcsvc.c:1842:nfs_rpcsvc_request_create] nfsrpc: RPC XID: 7188ef45, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2010-08-16 13:50:47.668296] D [rpcsvc.c:1285:nfs_rpcsvc_program_actor] nfsrpc: Actor found: NFS3 - ACCESS
[2010-08-16 13:50:47.668314] D [nfs3-helpers.c:2227:nfs3_log_common_call] nfs-nfsv3: XID: 7188ef45, ACCESS: args: FH: hashcount 6, xlid 0, gen 5506007464832682866, ino 9240768
[2010-08-16 13:50:47.668329] T [nfs3.c:1186:nfs3_access] nfs-nfsv3: FH to Volume: statprefetch
[2010-08-16 13:50:47.668354] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-16 13:50:47.668373] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-16 13:50:47.668392] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-16 13:50:47.668408] T [nfs-fops.c:326:nfs_fop_stat] nfs: Stat: /nfsalpha204/ip-10-244-167-207/test9/linux-2.6.35/Documentation/video4linux
[2010-08-16 13:50:47.668441] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891177888, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.668459] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 232, payload: 104, rpc hdr: 128
[2010-08-16 13:50:47.668488] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891177888, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.668507] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 232, payload: 104, rpc hdr: 128
[2010-08-16 13:50:47.668584] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891177888, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.668601] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 232, payload: 104, rpc hdr: 128
[2010-08-16 13:50:47.668635] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891177888, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.668652] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 232, payload: 104, rpc hdr: 128
[2010-08-16 13:50:47.668688] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x675380, readsize: 100, dataread:100
[2010-08-16 13:50:47.668713] T [rpcsvc.c:2360:nfs_rpcsvc_record_update_state] nfsrpc: Regular frag
[2010-08-16 13:50:47.668729] T [rpcsvc.c:1176:nfs_rpcsvc_record_read_complete_frag] nfsrpc: Fragment remaining: 0
[2010-08-16 13:50:47.668744] T [rpcsvc.c:2382:nfs_rpcsvc_record_update_state] nfsrpc: Full Record Received.
[2010-08-16 13:50:47.668760] T [rpcsvc-auth.c:247:nfs_rpcsvc_auth_request_init] nfsrpc: Auth handler: AUTH_NULL
[2010-08-16 13:50:47.668775] D [rpcsvc.c:1842:nfs_rpcsvc_request_create] nfsrpc: RPC XID: 90b1f7dd, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3
[2010-08-16 13:50:47.668791] D [rpcsvc.c:1285:nfs_rpcsvc_program_actor] nfsrpc: Actor found: NFS3 - LOOKUP
[2010-08-16 13:50:47.668809] D [nfs3-helpers.c:2239:nfs3_log_fh_entry_call] nfs-nfsv3: XID: 90b1f7dd, LOOKUP: args: FH: hashcount 9, xlid 0, gen 5506007464832793684, ino 40902936, name: regs-bbu.h
[2010-08-16 13:50:47.668826] T [nfs3.c:1040:nfs3_lookup] nfs-nfsv3: FH to Volume: statprefetch
[2010-08-16 13:50:47.668843] T [nfs3-helpers.c:2940:nfs3_fh_resolve_entry_hard] nfs-nfsv3: FH hard resolution: ino: 40902936, gen: 5506007464832793684, entry: regs-bbu.h, hashidx: 0
[2010-08-16 13:50:47.668869] T [nfs3-helpers.c:2948:nfs3_fh_resolve_entry_hard] nfs-nfsv3: Entry needs lookup: /nfsalpha101/ip-10-204-110-127/test8/linux-2.6.35/arch/arm/mach-ns9xxx/include/mach/regs-bbu.h
[2010-08-16 13:50:47.668886] T [nfs-fops.c:279:nfs_fop_lookup] nfs: Lookup: /nfsalpha101/ip-10-204-110-127/test8/linux-2.6.35/arch/arm/mach-ns9xxx/include/mach/regs-bbu.h
[2010-08-16 13:50:47.668943] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.668969] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 700, payload: 572, rpc hdr: 128
[2010-08-16 13:50:47.668998] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.669015] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 700, payload: 572, rpc hdr: 128
[2010-08-16 13:50:47.669053] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.669074] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 700, payload: 572, rpc hdr: 128
[2010-08-16 13:50:47.669104] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x675380, readsize: 4, dataread:4
[2010-08-16 13:50:47.669121] T [rpcsvc.c:1078:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Received fragment size: 76
[2010-08-16 13:50:47.669145] T [rpcsvc.c:1085:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Regular RPC header, remaining: 76
[2010-08-16 13:50:47.669163] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x70ff00, readsize: 4, dataread:4
[2010-08-16 13:50:47.669179] T [rpcsvc.c:1078:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Received fragment size: 112
[2010-08-16 13:50:47.669193] T [rpcsvc.c:1085:nfs_rpcsvc_record_read_complete_fraghdr] nfsrpc: Regular RPC header, remaining: 112
[2010-08-16 13:50:47.669211] T [rpcsvc.c:2423:nfs_rpcsvc_conn_data_poll_in] nfsrpc: conn: 0x70ff00, readsize: 112, dataread:112
[2010-08-16 13:50:47.669226] T [rpcsvc.c:2360:nfs_rpcsvc_record_update_state] nfsrpc: Regular frag
[2010-08-16 13:50:47.669240] T [rpcsvc.c:1176:nfs_rpcsvc_record_read_complete_frag] nfsrpc: Fragment remaining: 0
[2010-08-16 13:50:47.669254] T [rpcsvc.c:2382:nfs_rpcsvc_record_update_state] nfsrpc: Full Record Received.
[2010-08-16 13:50:47.669269] T [rpcsvc-auth.c:247:nfs_rpcsvc_auth_request_init] nfsrpc: Auth handler: AUTH_NULL
[2010-08-16 13:50:47.669284] D [rpcsvc.c:1842:nfs_rpcsvc_request_create] nfsrpc: RPC XID: 7288ef45, Ver: 2, Program: 100003, ProgVers: 3, Proc: 8
[2010-08-16 13:50:47.669299] D [rpcsvc.c:1285:nfs_rpcsvc_program_actor] nfsrpc: Actor found: NFS3 - CREATE
[2010-08-16 13:50:47.669344] D [nfs3-helpers.c:2278:nfs3_log_create_call] nfs-nfsv3: XID: 7288ef45, CREATE: args: FH: hashcount 9, xlid 0, gen 5506007464832803432, ino 44054400, name: reg_umi.h, mode: EXCLUSIVE
[2010-08-16 13:50:47.669368] T [nfs3.c:2228:nfs3_create] nfs-nfsv3: FH to Volume: statprefetch
[2010-08-16 13:50:47.669390] T [nfs3-helpers.c:2940:nfs3_fh_resolve_entry_hard] nfs-nfsv3: FH hard resolution: ino: 44054400, gen: 5506007464832803432, entry: reg_umi.h, hashidx: 0
pending frames:

patchset: v3.1.0qa5-15-gc87c868
signal received: 11
time of crash: 2010-08-16 13:50:47
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.0qa5-15-gc87c868
[2010-08-16 13:50:47.664927] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395485 Program: GlusterFS 3.1, ProgVers: 310, Proc: 4
[2010-08-16 13:50:47.670025] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2641883 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.670049] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.670072] D [nfs3-helpers.c:2398:nfs3_log_write_res] nfs-nfsv3: XID: 1588ef45, WRITE: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning), count: 1496, STABLE,wverf: 1281967445
[2010-08-16 13:50:47.670096] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 136
[2010-08-16 13:50:47.670114] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 160, payload: 136, rpc hdr: 24
[2010-08-16 13:50:47.670130] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 136
[2010-08-16 13:50:47.670149] T [nfs3.c:202:nfs3_call_state_wipe] nfs-nfsv3: fd ref: 3
[2010-08-16 13:50:47.670186] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209110 Program: GlusterFS 3.1, ProgVers: 310, Proc: 4
[2010-08-16 13:50:47.670209] T [dht-layout.c:355:dht_layout_merge] distribute: missing disk layout on mirror-3. err = -1
[2010-08-16 13:50:47.670240] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.670258] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.670289] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.670306] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.670354] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642035 Program: GlusterFS 3.1, ProgVers: 310, Proc: 13
[2010-08-16 13:50:47.670438] T [io-cache.c:133:ioc_inode_flush] iocache: locked inode(0x2aab00554640)
[2010-08-16 13:50:47.670458] T [io-cache.c:137:ioc_inode_flush] iocache: unlocked inode(0x2aab00554640)
[2010-08-16 13:50:47.670478] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-16 13:50:47.670502] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.670546] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891163200, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.670563] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 164, payload: 36, rpc hdr: 128
[2010-08-16 13:50:47.670588] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891163200, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.670604] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 164, payload: 36, rpc hdr: 128
[2010-08-16 13:50:47.670636] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891159936, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.670653] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 208, payload: 80, rpc hdr: 128
[2010-08-16 13:50:47.670685] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382054 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.670744] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740332 Program: GlusterFS 3.1, ProgVers: 310, Proc: 30
[2010-08-16 13:50:47.670781] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161348 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.670808] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775665 Program: GlusterFS 3.1, ProgVers: 310, Proc: 31
[2010-08-16 13:50:47.670852] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891163744, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.670869] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.670931] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891163744, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.670953] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.670997] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775663 Program: GlusterFS 3.1, ProgVers: 310, Proc: 4
[2010-08-16 13:50:47.671029] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161349 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.671060] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740334 Program: GlusterFS 3.1, ProgVers: 310, Proc: 31
[2010-08-16 13:50:47.671086] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891145792, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.671104] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.671155] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891145792, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.671173] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.671205] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891145792, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.671224] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.671270] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382056 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.671343] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642036 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.671378] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642037 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.671423] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382055 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.671453] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891150144, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.671480] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 276, payload: 148, rpc hdr: 128
[2010-08-16 13:50:47.671519] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740335 Program: GlusterFS 3.1, ProgVers: 310, Proc: 31
[2010-08-16 13:50:47.671595] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161350 Program: GlusterFS 3.1, ProgVers: 310, Proc: 38
[2010-08-16 13:50:47.671632] D [nfs3-helpers.c:2359:nfs3_log_common_res] nfs-nfsv3: XID: 6d88ef45, SETATTR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
[2010-08-16 13:50:47.671651] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 120
[2010-08-16 13:50:47.671667] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 144, payload: 120, rpc hdr: 24
[2010-08-16 13:50:47.671682] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 120
[2010-08-16 13:50:47.671704] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.671731] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775664 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.671762] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209114 Program: GlusterFS 3.1, ProgVers: 310, Proc: 31
[2010-08-16 13:50:47.671796] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775666 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.671822] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161351 Program: GlusterFS 3.1, ProgVers: 310, Proc: 38
[2010-08-16 13:50:47.671844] D [nfs3-helpers.c:2359:nfs3_log_common_res] nfs-nfsv3: XID: 8eb1f7dd, SETATTR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
[2010-08-16 13:50:47.671862] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 120
[2010-08-16 13:50:47.671883] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 144, payload: 120, rpc hdr: 24
[2010-08-16 13:50:47.671898] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 120
[2010-08-16 13:50:47.671936] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.671963] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740333 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.671998] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382058 Program: GlusterFS 3.1, ProgVers: 310, Proc: 4
[2010-08-16 13:50:47.672032] T [dht-layout.c:355:dht_layout_merge] distribute: missing disk layout on mirror-2. err = -1
[2010-08-16 13:50:47.672061] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.672078] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.672107] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.672123] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.672157] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.672191] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.672237] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642038 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.672283] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642039 Program: GlusterFS 3.1, ProgVers: 310, Proc: 1
[2010-08-16 13:50:47.672315] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382057 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.672341] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891180336, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.672358] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 276, payload: 148, rpc hdr: 128
[2010-08-16 13:50:47.672390] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740336 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.672430] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161352 Program: GlusterFS 3.1, ProgVers: 310, Proc: 1
[2010-08-16 13:50:47.672467] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775667 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.672501] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209089 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.672583] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775668 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.672617] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740337 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.672664] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382059 Program: GlusterFS 3.1, ProgVers: 310, Proc: 23
[2010-08-16 13:50:47.672698] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642040 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.672765] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209115 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.672812] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382061 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.672841] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740338 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.672884] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775669 Program: GlusterFS 3.1, ProgVers: 310, Proc: 1
[2010-08-16 13:50:47.672915] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395486 Program: GlusterFS 3.1, ProgVers: 310, Proc: 23
[2010-08-16 13:50:47.672940] T [ioc-inode.c:174:ioc_inode_update] iocache: locked table(0x6452a0)
[2010-08-16 13:50:47.672956] T [ioc-inode.c:182:ioc_inode_update] iocache: adding to inode_lru[1]
[2010-08-16 13:50:47.672970] T [ioc-inode.c:184:ioc_inode_update] iocache: unlocked table(0x6452a0)
[2010-08-16 13:50:47.672984] T [io-cache.c:614:ioc_create_cbk] iocache: locked inode(0x8e96170)
[2010-08-16 13:50:47.672998] T [io-cache.c:626:ioc_create_cbk] iocache: unlocked inode(0x8e96170)
[2010-08-16 13:50:47.673040] D [nfs3-helpers.c:2412:nfs3_log_newfh_res] nfs-nfsv3: XID: 5d88ef45, CREATE: NFS: 0(Call completed successfully.), POSIX: 0(Success), FH: hashcount 10, xlid 0, gen 5506008053243323339, ino 95961050
[2010-08-16 13:50:47.673074] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 152
[2010-08-16 13:50:47.673091] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 176, payload: 152, rpc hdr: 24
[2010-08-16 13:50:47.673109] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 152
[2010-08-16 13:50:47.673144] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673161] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 484, payload: 356, rpc hdr: 128
[2010-08-16 13:50:47.673192] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673209] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 484, payload: 356, rpc hdr: 128
[2010-08-16 13:50:47.673239] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673256] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 484, payload: 356, rpc hdr: 128
[2010-08-16 13:50:47.673295] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775670 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.673363] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740339 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.673397] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382062 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.673422] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209113 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.673486] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209116 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.673541] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673558] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 880, payload: 752, rpc hdr: 128
[2010-08-16 13:50:47.673600] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673625] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 880, payload: 752, rpc hdr: 128
[2010-08-16 13:50:47.673682] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673698] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1084, payload: 956, rpc hdr: 128
[2010-08-16 13:50:47.673738] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673754] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1084, payload: 956, rpc hdr: 128
[2010-08-16 13:50:47.673821] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673838] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1288, payload: 1160, rpc hdr: 128
[2010-08-16 13:50:47.673874] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673890] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1288, payload: 1160, rpc hdr: 128
[2010-08-16 13:50:47.673923] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.673940] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1288, payload: 1160, rpc hdr: 128
[2010-08-16 13:50:47.674002] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.674018] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1288, payload: 1160, rpc hdr: 128
[2010-08-16 13:50:47.674072] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.674099] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1288, payload: 1160, rpc hdr: 128
[2010-08-16 13:50:47.674137] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382063 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.674171] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740340 Program: GlusterFS 3.1, ProgVers: 310, Proc: 38
[2010-08-16 13:50:47.674213] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395487 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.674244] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395488 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.674276] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382064 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.674303] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209119 Program: GlusterFS 3.1, ProgVers: 310, Proc: 30
[2010-08-16 13:50:47.674337] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209117 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.674362] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891188496, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.674378] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 276, payload: 148, rpc hdr: 128
[2010-08-16 13:50:47.674411] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382065 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.674441] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382066 Program: GlusterFS 3.1, ProgVers: 310, Proc: 38
[2010-08-16 13:50:47.674465] D [nfs3-helpers.c:2359:nfs3_log_common_res] nfs-nfsv3: XID: 6f88ef45, SETATTR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
[2010-08-16 13:50:47.674492] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 120
[2010-08-16 13:50:47.674509] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 144, payload: 120, rpc hdr: 24
[2010-08-16 13:50:47.674524] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 120
[2010-08-16 13:50:47.674545] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.674574] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382067 Program: GlusterFS 3.1, ProgVers: 310, Proc: 38
[2010-08-16 13:50:47.674603] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382053 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.674663] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.674735] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 880, payload: 752, rpc hdr: 128
[2010-08-16 13:50:47.674796] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.674822] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 880, payload: 752, rpc hdr: 128
[2010-08-16 13:50:47.675425] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.675443] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1084, payload: 956, rpc hdr: 128
[2010-08-16 13:50:47.675492] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.675511] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1084, payload: 956, rpc hdr: 128
[2010-08-16 13:50:47.675573] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.675592] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1084, payload: 956, rpc hdr: 128
[2010-08-16 13:50:47.675627] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.675646] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1084, payload: 956, rpc hdr: 128
[2010-08-16 13:50:47.675686] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.675709] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1084, payload: 956, rpc hdr: 128
[2010-08-16 13:50:47.675770] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.675791] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1288, payload: 1160, rpc hdr: 128
[2010-08-16 13:50:47.675834] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891174352, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.675857] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1288, payload: 1160, rpc hdr: 128
[2010-08-16 13:50:47.675926] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209120 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.675964] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740341 Program: GlusterFS 3.1, ProgVers: 310, Proc: 38
[2010-08-16 13:50:47.675996] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775671 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.676027] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395489 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.676064] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382069 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.676102] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395491 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.676137] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740342 Program: GlusterFS 3.1, ProgVers: 310, Proc: 1
[2010-08-16 13:50:47.676171] D [nfs3-helpers.c:2359:nfs3_log_common_res] nfs-nfsv3: XID: 7188ef45, ACCESS: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
[2010-08-16 13:50:47.676191] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 96
[2010-08-16 13:50:47.676208] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 120, payload: 96, rpc hdr: 24
[2010-08-16 13:50:47.676239] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 96
[2010-08-16 13:50:47.676286] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382070 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.676321] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740343 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.676354] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395490 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.676379] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891167824, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.676413] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 276, payload: 148, rpc hdr: 128
[2010-08-16 13:50:47.676460] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395492 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.676529] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.676548] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 892, payload: 764, rpc hdr: 128
[2010-08-16 13:50:47.676587] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.676608] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 892, payload: 764, rpc hdr: 128
[2010-08-16 13:50:47.676657] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.676674] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1096, payload: 968, rpc hdr: 128
[2010-08-16 13:50:47.676727] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.676749] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1096, payload: 968, rpc hdr: 128
[2010-08-16 13:50:47.676802] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.676822] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1096, payload: 968, rpc hdr: 128
[2010-08-16 13:50:47.676853] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.676870] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1096, payload: 968, rpc hdr: 128
[2010-08-16 13:50:47.676902] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.676919] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1096, payload: 968, rpc hdr: 128
[2010-08-16 13:50:47.676972] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.677002] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1300, payload: 1172, rpc hdr: 128
[2010-08-16 13:50:47.677040] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.677061] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1300, payload: 1172, rpc hdr: 128
[2010-08-16 13:50:47.677099] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740344 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.678823] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382068 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.678865] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642042 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.678897] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161353 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.678950] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775672 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.678978] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382072 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.679007] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382073 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.679034] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3775673 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.679061] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161354 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.679098] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642043 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.679135] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382074 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.680399] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382071 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.680433] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740345 Program: GlusterFS 3.1, ProgVers: 310, Proc: 31
[2010-08-16 13:50:47.680481] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642044 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.680515] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161355 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.680560] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4161339 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.680598] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740347 Program: GlusterFS 3.1, ProgVers: 310, Proc: 31
[2010-08-16 13:50:47.680652] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740351 Program: GlusterFS 3.1, ProgVers: 310, Proc: 31
[2010-08-16 13:50:47.680697] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382060 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.680880] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740346 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.681138] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642019 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.681196] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.681236] D [nfs3-helpers.c:2398:nfs3_log_write_res] nfs-nfsv3: XID: 6688ef45, WRITE: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning), count: 15066, STABLE,wverf: 1281967445
[2010-08-16 13:50:47.681660] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 136
[2010-08-16 13:50:47.681687] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 160, payload: 136, rpc hdr: 24
[2010-08-16 13:50:47.681705] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 136
[2010-08-16 13:50:47.681731] T [nfs3.c:202:nfs3_call_state_wipe] nfs-nfsv3: fd ref: 3
[2010-08-16 13:50:47.681777] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2493037 Program: GlusterFS 3.1, ProgVers: 310, Proc: 4
[2010-08-16 13:50:47.681804] T [dht-layout.c:355:dht_layout_merge] distribute: missing disk layout on mirror-1. err = -1
[2010-08-16 13:50:47.681839] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.681860] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.681916] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.681946] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 468, payload: 340, rpc hdr: 128
[2010-08-16 13:50:47.681986] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209121 Program: GlusterFS 3.1, ProgVers: 310, Proc: 31
[2010-08-16 13:50:47.682051] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740348 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.682090] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2382046 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.682133] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740349 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.682165] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209118 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.682205] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209122 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.682274] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740350 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.682319] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 3740352 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.682488] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 4209123 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.683744] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2493039 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.686670] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395494 Program: GlusterFS 3.1, ProgVers: 310, Proc: 38
[2010-08-16 13:50:47.686723] D [nfs3-helpers.c:2359:nfs3_log_common_res] nfs-nfsv3: XID: 7088ef45, SETATTR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning)
[2010-08-16 13:50:47.686748] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 120
[2010-08-16 13:50:47.686767] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 144, payload: 120, rpc hdr: 24
[2010-08-16 13:50:47.686783] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 120
[2010-08-16 13:50:47.686853] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2631866 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.686879] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.686900] D [nfs3-helpers.c:2398:nfs3_log_write_res] nfs-nfsv3: XID: 81b1f7dd, WRITE: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning), count: 94, STABLE,wverf: 1281967445
[2010-08-16 13:50:47.686922] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 136
[2010-08-16 13:50:47.686940] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 160, payload: 136, rpc hdr: 24
[2010-08-16 13:50:47.686956] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 136
[2010-08-16 13:50:47.686977] T [nfs3.c:202:nfs3_call_state_wipe] nfs-nfsv3: fd ref: 3
[2010-08-16 13:50:47.687014] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2493042 Program: GlusterFS 3.1, ProgVers: 310, Proc: 33
[2010-08-16 13:50:47.687049] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891154496, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.687067] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 272, payload: 144, rpc hdr: 128
[2010-08-16 13:50:47.687097] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891154496, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.687115] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 272, payload: 144, rpc hdr: 128
[2010-08-16 13:50:47.687171] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2642041 Program: GlusterFS 3.1, ProgVers: 310, Proc: 16
[2010-08-16 13:50:47.687197] T [write-behind.c:440:wb_sync] writebehind: no vectors are to besynced
[2010-08-16 13:50:47.687219] D [nfs3-helpers.c:2398:nfs3_log_write_res] nfs-nfsv3: XID: 8ab1f7dd, WRITE: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning), count: 2383, STABLE,wverf: 1281967445
[2010-08-16 13:50:47.687238] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 136
[2010-08-16 13:50:47.687272] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 160, payload: 136, rpc hdr: 24
[2010-08-16 13:50:47.687291] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 136
[2010-08-16 13:50:47.687311] T [nfs3.c:202:nfs3_call_state_wipe] nfs-nfsv3: fd ref: 3
[2010-08-16 13:50:47.687356] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2493043 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.687397] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2631867 Program: GlusterFS 3.1, ProgVers: 310, Proc: 17
[2010-08-16 13:50:47.687446] D [nfs3-helpers.c:2412:nfs3_log_newfh_res] nfs-nfsv3: XID: 5088ef45, MKDIR: NFS: 0(Call completed successfully.), POSIX: 117(Structure needs cleaning), FH: hashcount 6, xlid 0, gen 5506007464832804132, ino 44434104
[2010-08-16 13:50:47.687470] T [rpcsvc.c:1594:nfs_rpcsvc_submit_generic] nfsrpc: Tx message: 144
[2010-08-16 13:50:47.687487] T [rpcsvc.c:1384:nfs_rpcsvc_record_build_header] nfsrpc: Reply fraglen 168, payload: 144, rpc hdr: 24
[2010-08-16 13:50:47.687503] T [rpcsvc.c:1421:nfs_rpcsvc_conn_submit] nfsrpc: Tx Header: 28, payload: 144
[2010-08-16 13:50:47.687545] T [rpc-clnt.c:619:rpc_clnt_reply_init] rpc-clnt: RPC XID: 2395495 Program: GlusterFS 3.1, ProgVers: 310, Proc: 27
[2010-08-16 13:50:47.687610] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.687630] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 904, payload: 776, rpc hdr: 128
[2010-08-16 13:50:47.687673] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.687696] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 904, payload: 776, rpc hdr: 128
[2010-08-16 13:50:47.687774] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.687792] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1108, payload: 980, rpc hdr: 128
[2010-08-16 13:50:47.687824] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.687864] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1108, payload: 980, rpc hdr: 128
[2010-08-16 13:50:47.687913] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.687932] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1108, payload: 980, rpc hdr: 128
[2010-08-16 13:50:47.687973] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.687991] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1108, payload: 980, rpc hdr: 128
[2010-08-16 13:50:47.688029] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.688050] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1108, payload: 980, rpc hdr: 128
[2010-08-16 13:50:47.688116] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.688136] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1312, payload: 1184, rpc hdr: 128
[2010-08-16 13:50:47.688196] T [rpc-clnt.c:1187:rpc_clnt_record] : Auth Info: pid: 2891155584, uid: 0, gid: 0, owner: 2
[2010-08-16 13:50:47.688219] T [rpc-clnt.c:1079:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 1312, payload: 1184, rpc hdr: 128
/lib64/libc.so.6[0x2aaaab7abf30]
/opt/glusterfs/3.1.0qa6/lib/libglusterfs.so.0[0x2aaaaacefaff]
/opt/glusterfs/3.1.0qa6/lib/libglusterfs.so.0[0x2aaaaacefac7]
/opt/glusterfs/3.1.0qa6/lib/libglusterfs.so.0[0x2aaaaacefa10]
/opt/glusterfs/3.1.0qa6/lib/libglusterfs.so.0(inode_unref+0x58)[0x2aaaaacee4f6]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs_entry_loc_fill+0x152)[0x2aaaad91994e]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs3_fh_resolve_entry_hard+0x105)[0x2aaaad93c015]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs3_fh_resolve_entry+0x2b)[0x2aaaad93c232]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs3_fh_resolve_and_resume+0xf6)[0x2aaaad93c37b]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs3_create+0x352)[0x2aaaad92c208]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs3svc_create+0x124)[0x2aaaad92c3a4]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs_rpcsvc_handle_rpc_call+0x9b)[0x2aaaad942aec]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs_rpcsvc_record_update_state+0x1d0)[0x2aaaad9437ee]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs_rpcsvc_conn_data_poll_in+0xd5)[0x2aaaad943944]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs_rpcsvc_conn_data_handler+0x87)[0x2aaaad943d68]
/opt/glusterfs/3.1.0qa6/lib/libglusterfs.so.0[0x2aaaaad02145]
/opt/glusterfs/3.1.0qa6/lib/libglusterfs.so.0[0x2aaaaad02318]
/opt/glusterfs/3.1.0qa6/lib/libglusterfs.so.0(event_dispatch+0x65)[0x2aaaaad025e8]
/opt/glusterfs/3.1.0qa6/lib/glusterfs/3.1.0qa5-15-gc87c868/xlator/nfs/server.so(nfs_rpcsvc_stage_proc+0x32)[0x2aaaad93f236]
/lib64/libpthread.so.0[0x2aaaab566407]
/lib64/libc.so.6(clone+0x6d)[0x2aaaab84fb0d]
---------
==============================

(gdb) bt full
#0  0x00002aaaaacefaff in __list_splice (list=0x2aab00565e80, head=0x40affc00) at list.h:105
No locals.
#1  0x00002aaaaacefac7 in list_splice_init (list=0x2aab00565e80, head=0x40affc00) at list.h:129
No locals.
#2  0x00002aaaaacefa10 in inode_table_prune (table=0x2aab00565df0) at inode.c:1146
	ret = 0
	purge = {next = 0x40affc00, prev = 0x40affc00}
	del = (inode_t *) 0x0
	tmp = (inode_t *) 0x0
	entry = (inode_t *) 0x0
#3  0x00002aaaaacee4f6 in inode_unref (inode=0x2aab00462b90) at inode.c:462
	table = (inode_table_t *) 0x2aab00565df0
#4  0x00002aaaad91994e in nfs_entry_loc_fill (itable=0x644e90, ino=44054400, gen=5506007464832803432, entry=0x2aaadf57cad0 "reg_umi.h", loc=0x2aaaaac51e48, how=2)
    at nfs-common.c:379
	parent = (inode_t *) 0x2aaafc5c96e0
	entryinode = (inode_t *) 0x2aab00462b90
	ret = -2
	resolvedpath = 0x0
	pret = 0
#5  0x00002aaaad93c015 in nfs3_fh_resolve_entry_hard (cs=0x2aaaaac51a60) at nfs3-helpers.c:2942
	ret = -14
	nfu = {uid = 0, gids = {0 <repeats 17 times>}, ngrps = 1}
	__FUNCTION__ = "nfs3_fh_resolve_entry_hard"
#6  0x00002aaaad93c232 in nfs3_fh_resolve_entry (cs=0x2aaaaac51a60) at nfs3-helpers.c:2995
	ret = -14
#7  0x00002aaaad93c37b in nfs3_fh_resolve_and_resume (cs=0x2aaaaac51a60, fh=0x40affed0, entry=0x40afff10 "reg_umi.h", resum_fn=0x2aaaad92bdac <nfs3_create_resume>)
    at nfs3-helpers.c:3022
	ret = -14
#8  0x00002aaaad92c208 in nfs3_create (req=0x763390, dirfh=0x40affed0, name=0x40afff10 "reg_umi.h", mode=EXCLUSIVE, sattr=0x40affe90, cverf=78052738690983)
    at nfs3.c:2238
	vol = (xlator_t *) 0x642850
	stat = NFS3ERR_SERVERFAULT
	ret = -14
	nfs3 = (struct nfs3_state *) 0x66a5d0
	cs = (nfs3_call_state_t *) 0x2aaaaac51a60
	__FUNCTION__ = "nfs3_create"
#9  0x00002aaaad92c3a4 in nfs3svc_create (req=0x763390) at nfs3.c:2275
	name = "reg_umi.h\000\000��*\000\000\030\000\000\0000\000\000\000\000\000�@\000\000\000\000@��@", '\0' <repeats 12 times>, "�\221b\000\000\000\000\000 \000\000��*\000\000���@", '\0' <repeats 21 times>, " �@\000\000\000\000\000\020\000\000\000\000\000\000,\017\177��*\000\000\000\020\000\000\000\000\000\000 \000\000��*\000\000�\002�@", '\0' <repeats 21 times>, " �@\000\000\000\000\000\020\000\000\000\000\000\000,\017\177��*\000\000����", '\0' <repeats 28 times>, "P�b", '\0' <repeats 13 times>, "�\002�@\000\000\000\000\021\017�*\000\000�Y���"...
	dirfh = {ident = ":O", hashcount = 9, xlatorid = 0, gen = 5506007464832803432, ino = 44054400, entryhash = {257, 28875, 43295, 37074, 18372, 18855, 21157, 
    43858, 55885, 0 <repeats 12 times>}}
	args = {where = {dir = {data = {data_len = 40, data_val = 0x40affed0 ":O\t"}}, name = 0x40afff10 "reg_umi.h"}, how = {mode = EXCLUSIVE, createhow3_u = {
      obj_attributes = {mode = {set_it = 298020775, set_mode3_u = {mode = 18173}}, uid = {set_it = 0, set_uid3_u = {uid = 0}}, gid = {set_it = 0, set_gid3_u = {
            gid = 0}}, size = {set_it = 0, set_size3_u = {size = 0}}, atime = {set_it = DONT_CHANGE, set_atime_u = {atime = {seconds = 0, nseconds = 0}}}, mtime = {
          set_it = DONT_CHANGE, set_mtime_u = {mtime = {seconds = 0, nseconds = 0}}}}, verf = "�o�\021�F\000"}}}
	ret = -1
	cverf = 78052738690983
	__FUNCTION__ = "nfs3svc_create"
#10 0x00002aaaad942aec in nfs_rpcsvc_handle_rpc_call (conn=0x70ff00) at ../../../../xlators/nfs/lib/src//rpcsvc.c:1901
	actor = (rpcsvc_actor_t *) 0x2aaaadb5b100
	req = (rpcsvc_request_t *) 0x763390
	ret = -1
#11 0x00002aaaad9437ee in nfs_rpcsvc_record_update_state (conn=0x70ff00, dataread=0) at ../../../../xlators/nfs/lib/src//rpcsvc.c:2383
	rs = (rpcsvc_record_state_t *) 0x70ff18
	svc = (rpcsvc_t *) 0x0
	__FUNCTION__ = "nfs_rpcsvc_record_update_state"
#12 0x00002aaaad943944 in nfs_rpcsvc_conn_data_poll_in (conn=0x70ff00) at ../../../../xlators/nfs/lib/src//rpcsvc.c:2426
	dataread = 112
	readsize = 112
	readaddr = 0x2aab04c20000 "r\210�E"
	ret = -1
	__FUNCTION__ = "nfs_rpcsvc_conn_data_poll_in"
#13 0x00002aaaad943d68 in nfs_rpcsvc_conn_data_handler (fd=22, idx=4, data=0x70ff00, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../xlators/nfs/lib/src//rpcsvc.c:2555
	conn = (rpcsvc_conn_t *) 0x70ff00
	ret = 0
#14 0x00002aaaaad02145 in event_dispatch_epoll_handler (event_pool=0x644b80, events=0x65f1a0, i=0) at event.c:812
	event_data = (struct event_data *) 0x65f1a4
	handler = (event_handler_t) 0x2aaaad943ce1 <nfs_rpcsvc_conn_data_handler>
	data = (void *) 0x70ff00
	idx = 4
	ret = -1
	__FUNCTION__ = "event_dispatch_epoll_handler"
#15 0x00002aaaaad02318 in event_dispatch_epoll (event_pool=0x644b80) at event.c:876
	events = (struct epoll_event *) 0x65f1a0
	size = 2
	i = 0
	ret = 2
	__FUNCTION__ = "event_dispatch_epoll"
#16 0x00002aaaaad025e8 in event_dispatch (event_pool=0x644b80) at event.c:984
	ret = -1
	__FUNCTION__ = "event_dispatch"
#17 0x00002aaaad93f236 in nfs_rpcsvc_stage_proc (arg=0x62b2a0) at ../../../../xlators/nfs/lib/src//rpcsvc.c:64
	stg = (rpcsvc_stage_t *) 0x62b2a0
#18 0x00002aaaab566407 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#19 0x00002aaaab84fb0d in clone () from /lib64/libc.so.6
No symbol table info available.
Comment 1 Shehjar Tikoo 2010-09-03 03:50:44 UTC 
*** Bug 1358 has been marked as a duplicate of this bug. ***
Comment 2 Shehjar Tikoo 2010-10-05 08:18:10 UTC 
Not sure whats going on here. Havent run into this in weeks of 4x3 dist-repl tests with  nfs. Closing.