Bug 763085 (GLUSTER-1353)

Summary: gnfs crash on nfs3_fill_create3res
Product: [Community] GlusterFS Reporter: Lakshmipathi G <lakshmipathi>
Component: nfsAssignee: Amar Tumballi <amarts>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: low    
Version: 3.1-alphaCC: gluster-bugs, vraman
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: nfs
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Shehjar Tikoo 2010-08-13 03:25:07 UTC 
The bug is in protocol/client or protocol/server because the op_ret being returned is -1 but op_errno is not set.



#####################################################################
### NFS frames come after this one #############################
#####################################################################
#7  0x00002aaaac569379 in afr_fresh_lookup_cbk (frame=0x7bafd20, cookie=0x1,
this=0x63b940, op_ret=-1, op_errno=0, inode=0x7bafdc0, buf=0x7fff2a33d640,
xattr=0x0, 
    postparent=0x7fff2a33d5d0) at afr.c:842
    local = (afr_local_t *) 0x7bec940
    priv = (afr_private_t *) 0x63fb40
    lookup_buf = (struct iatt *) 0x7beca88
    call_count = 0
    child_index = 1
    first_up_child = -1
#8  0x00002aaaac34b6fa in client3_1_lookup_cbk (req=0x2aaaad732da0,
iov=0x7fff2a33d840, count=1, myframe=0x7be0f70) at client3_1-fops.c:1955
    fn = (fop_lookup_cbk_t) 0x2aaaac568e44 <afr_fresh_lookup_cbk>
    _parent = (call_frame_t *) 0x7bafd20
    old_THIS = (xlator_t *) 0x632360
    local = (clnt_local_t *) 0x7be0ff0
    frame = (call_frame_t *) 0x7be0f70
    ret = 0
---Type <return> to continue, or q <return> to quit---
    rsp = {gfs_id = 0, op_ret = -1, op_errno = 0, stat = {ia_ino = 0, ia_gen =
Comment 1 Anand Avati 2010-08-13 04:56:58 UTC 
PATCH: http://patches.gluster.com/patch/4108 in master (protocol/client: send proper errno when op_ret is -1)
Comment 2 Lakshmipathi G 2010-08-13 06:07:07 UTC 
while running mixed test with 4x3 distributed-replicate  gnfs crashed.log and core file content are below - 

# tail -200 gNFS.log
[2010-08-12 17:48:26.392930] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.392946] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 168, payload: 40, rpc hdr: 128
[2010-08-12 17:48:26.392987] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.393008] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 168, payload: 40, rpc hdr: 128
[2010-08-12 17:48:26.393050] D [name.c:155:client_fill_address_family] ec2-75-101-206-122.compute-1.amazonaws.com-3: address-family not specified, guessing it to be inet/inet6
[2010-08-12 17:48:26.393071] T [common-utils.c:102:gf_resolve_ip6] resolver: flushing DNS cache
[2010-08-12 17:48:26.393087] T [common-utils.c:109:gf_resolve_ip6] resolver: DNS cache not present, freshly probing hostname: ec2-75-101-206-122.compute-1.amazonaws.com
[2010-08-12 17:48:26.423027] T [common-utils.c:150:gf_resolve_ip6] resolver: returning ip-10.196.170.96 (port-6677) for hostname: ec2-75-101-206-122.compute-1.amazonaws.com and port: 6677
[2010-08-12 17:48:26.423095] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.423123] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 168, payload: 40, rpc hdr: 128
[2010-08-12 17:48:26.423143] D [rpc-clnt.c:1287:rpc_clnt_submit] rpc-clnt: transmission of rpc-request failed
[2010-08-12 17:48:26.423173] D [name.c:155:client_fill_address_family] ec2-75-101-206-122.compute-1.amazonaws.com-1: address-family not specified, guessing it to be inet/inet6
[2010-08-12 17:48:26.423194] T [common-utils.c:102:gf_resolve_ip6] resolver: flushing DNS cache
[2010-08-12 17:48:26.423212] T [common-utils.c:109:gf_resolve_ip6] resolver: DNS cache not present, freshly probing hostname: ec2-75-101-206-122.compute-1.amazonaws.com
[2010-08-12 17:48:26.431162] T [common-utils.c:150:gf_resolve_ip6] resolver: returning ip-10.196.170.96 (port-6675) for hostname: ec2-75-101-206-122.compute-1.amazonaws.com and port: 6675
[2010-08-12 17:48:26.431267] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.431294] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 168, payload: 40, rpc hdr: 128
[2010-08-12 17:48:26.431313] D [rpc-clnt.c:1287:rpc_clnt_submit] rpc-clnt: transmission of rpc-request failed
[2010-08-12 17:48:26.431342] T [socket.c:1689:socket_connect] ec2-75-101-206-122.compute-1.amazonaws.com-2: connect () called on transport already connected
[2010-08-12 17:48:26.431359] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.431375] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 168, payload: 40, rpc hdr: 128
[2010-08-12 17:48:26.431390] D [rpc-clnt.c:1287:rpc_clnt_submit] rpc-clnt: transmission of rpc-request failed
[2010-08-12 17:48:26.431410] D [dht-common.c:994:dht_attr_cbk] distribute: subvolume mirror-2 returned -1 (Transport endpoint is not connected)
[2010-08-12 17:48:26.431452] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.431469] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 168, payload: 40, rpc hdr: 128
[2010-08-12 17:48:26.431524] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 4, dataread: 4
[2010-08-12 17:48:26.431542] T [rpcsvc.c:1064:rpcsvc_record_read_complete_fraghdr] rpc-service: Received fragment size: 88
[2010-08-12 17:48:26.431571] T [rpcsvc.c:1071:rpcsvc_record_read_complete_fraghdr] rpc-service: Regular RPC header, remaining: 88
[2010-08-12 17:48:26.431591] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 88, dataread: 88
[2010-08-12 17:48:26.431606] T [rpcsvc.c:2333:rpcsvc_record_update_state] rpc-service: Regular frag
[2010-08-12 17:48:26.431621] T [rpcsvc.c:1158:rpcsvc_record_read_complete_frag] rpc-service: Fragment remaining: 0
[2010-08-12 17:48:26.431636] T [rpcsvc.c:2355:rpcsvc_record_update_state] rpc-service: Full Record Received.
[2010-08-12 17:48:26.431655] T [rpcsvc-auth.c:247:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_NULL
[2010-08-12 17:48:26.431670] D [rpcsvc.c:1817:rpcsvc_request_create] rpc-service: RPC XID: 2f14aa67, Ver: 2, Program: 100003, ProgVers: 3, Proc: 6
[2010-08-12 17:48:26.431687] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - READ
[2010-08-12 17:48:26.431707] D [nfs3-helpers.c:2343:nfs3_log_rw_call] nfs-nfsv3: XID: 2f14aa67, READ: args: FH: hashcount 4, xlid 0, gen 5504105043428906697, ino 657065232, offset: 59674624,  count: 4096
[2010-08-12 17:48:26.431724] T [nfs3.c:1514:nfs3_read] nfs-nfsv3: FH to Volume: distribute
[2010-08-12 17:48:26.431741] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-12 17:48:26.431760] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-12 17:48:26.431780] T [nfs3-helpers.c:2192:nfs3_file_open_and_resume] nfs-nfsv3: Opening: /nfsalpha152/ip-10-244-167-207/test15/fi4
[2010-08-12 17:48:26.431813] T [nfs3-helpers.c:2172:nfs3_fdcache_getfd] nfs-nfsv3: fd found in state: 2
[2010-08-12 17:48:26.431830] T [nfs3-helpers.c:1880:__nfs3_fdcache_update_entry] nfs-nfsv3: Updating fd: 0x2aaab422d370
[2010-08-12 17:48:26.431847] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-12 17:48:26.431882] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.431900] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 172, payload: 44, rpc hdr: 128
[2010-08-12 17:48:26.431942] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 4, dataread: 4
[2010-08-12 17:48:26.431982] T [rpcsvc.c:1064:rpcsvc_record_read_complete_fraghdr] rpc-service: Received fragment size: 80
[2010-08-12 17:48:26.431999] T [rpcsvc.c:1071:rpcsvc_record_read_complete_fraghdr] rpc-service: Regular RPC header, remaining: 80
[2010-08-12 17:48:26.432020] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 80, dataread: 80
[2010-08-12 17:48:26.432036] T [rpcsvc.c:2333:rpcsvc_record_update_state] rpc-service: Regular frag
[2010-08-12 17:48:26.432052] T [rpcsvc.c:1158:rpcsvc_record_read_complete_frag] rpc-service: Fragment remaining: 0
[2010-08-12 17:48:26.432067] T [rpcsvc.c:2355:rpcsvc_record_update_state] rpc-service: Full Record Received.
[2010-08-12 17:48:26.432084] T [rpcsvc-auth.c:247:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_NULL
[2010-08-12 17:48:26.432099] D [rpcsvc.c:1817:rpcsvc_request_create] rpc-service: RPC XID: 3014aa67, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2010-08-12 17:48:26.432116] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - ACCESS
[2010-08-12 17:48:26.432134] D [nfs3-helpers.c:2227:nfs3_log_common_call] nfs-nfsv3: XID: 3014aa67, ACCESS: args: FH: hashcount 5, xlid 0, gen 5504105043429226684, ino 305726016
[2010-08-12 17:48:26.432150] T [nfs3.c:1183:nfs3_access] nfs-nfsv3: FH to Volume: distribute
[2010-08-12 17:48:26.432165] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-12 17:48:26.432183] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-12 17:48:26.432202] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-12 17:48:26.432216] T [nfs-fops.c:326:nfs_fop_stat] nfs: Stat: /nfsalpha152/ip-10-244-167-207/test5/fileop_L1_0/fileop_L1_0_L2_36
[2010-08-12 17:48:26.432244] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.432261] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 224, payload: 96, rpc hdr: 128
[2010-08-12 17:48:26.432302] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.432323] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 224, payload: 96, rpc hdr: 128
[2010-08-12 17:48:26.432360] T [socket.c:1689:socket_connect] ec2-75-101-206-122.compute-1.amazonaws.com-2: connect () called on transport already connected
[2010-08-12 17:48:26.432380] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.432395] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 224, payload: 96, rpc hdr: 128
[2010-08-12 17:48:26.432410] D [rpc-clnt.c:1287:rpc_clnt_submit] rpc-clnt: transmission of rpc-request failed
[2010-08-12 17:48:26.432426] T [client3_1-fops.c:2576:client3_1_stat] ec2-75-101-206-122.compute-1.amazonaws.com-1: STAT 305726016 (/nfsalpha152/ip-10-244-167-207/test5/fileop_L1_0/fileop_L1_0_L2_36): failed to get remote inode number
[2010-08-12 17:48:26.432444] T [client3_1-fops.c:2576:client3_1_stat] ec2-75-101-206-122.compute-1.amazonaws.com-3: STAT 305726016 (/nfsalpha152/ip-10-244-167-207/test5/fileop_L1_0/fileop_L1_0_L2_36): failed to get remote inode number
[2010-08-12 17:48:26.432461] D [dht-common.c:994:dht_attr_cbk] distribute: subvolume mirror-2 returned -1 (Stale NFS file handle)
[2010-08-12 17:48:26.432484] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.432501] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 224, payload: 96, rpc hdr: 128
[2010-08-12 17:48:26.432528] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 4, dataread: 4
[2010-08-12 17:48:26.432544] T [rpcsvc.c:1064:rpcsvc_record_read_complete_fraghdr] rpc-service: Received fragment size: 128
[2010-08-12 17:48:26.432559] T [rpcsvc.c:1071:rpcsvc_record_read_complete_fraghdr] rpc-service: Regular RPC header, remaining: 128
[2010-08-12 17:48:26.432577] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 128, dataread: 128
[2010-08-12 17:48:26.432601] T [rpcsvc.c:2333:rpcsvc_record_update_state] rpc-service: Regular frag
[2010-08-12 17:48:26.432617] T [rpcsvc.c:1158:rpcsvc_record_read_complete_frag] rpc-service: Fragment remaining: 0
[2010-08-12 17:48:26.432631] T [rpcsvc.c:2355:rpcsvc_record_update_state] rpc-service: Full Record Received.
[2010-08-12 17:48:26.432646] T [rpcsvc-auth.c:247:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_NULL
[2010-08-12 17:48:26.432660] D [rpcsvc.c:1817:rpcsvc_request_create] rpc-service: RPC XID: 3114aa67, Ver: 2, Program: 100003, ProgVers: 3, Proc: 9
[2010-08-12 17:48:26.432675] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - MKDIR
[2010-08-12 17:48:26.432694] D [nfs3-helpers.c:2239:nfs3_log_fh_entry_call] nfs-nfsv3: XID: 3114aa67, MKDIR: args: FH: hashcount 4, xlid 0, gen 5504105043429226923, ino 657666120, name: fileop_dir_0_36_1
[2010-08-12 17:48:26.432711] T [nfs3.c:2428:nfs3_mkdir] nfs-nfsv3: FH to Volume: distribute
[2010-08-12 17:48:26.432728] T [nfs3-helpers.c:2940:nfs3_fh_resolve_entry_hard] nfs-nfsv3: FH hard resolution: ino: 657666120, gen: 5504105043429226923, entry: fileop_dir_0_36_1, hashidx: 0
[2010-08-12 17:48:26.432753] T [nfs3-helpers.c:2948:nfs3_fh_resolve_entry_hard] nfs-nfsv3: Entry needs lookup: /nfsalpha151/ip-10-244-167-207/test5/fileop_L1_0_L2_36/fileop_dir_0_36_1
[2010-08-12 17:48:26.432769] T [nfs-fops.c:279:nfs_fop_lookup] nfs: Lookup: /nfsalpha151/ip-10-244-167-207/test5/fileop_L1_0_L2_36/fileop_dir_0_36_1
[2010-08-12 17:48:26.432829] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.432847] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 652, payload: 524, rpc hdr: 128
[2010-08-12 17:48:26.432892] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.432911] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 652, payload: 524, rpc hdr: 128
[2010-08-12 17:48:26.432942] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.432959] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 652, payload: 524, rpc hdr: 128
[2010-08-12 17:48:26.432998] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 4, dataread: 4
[2010-08-12 17:48:26.433015] T [rpcsvc.c:1064:rpcsvc_record_read_complete_fraghdr] rpc-service: Received fragment size: 396
[2010-08-12 17:48:26.433030] T [rpcsvc.c:1071:rpcsvc_record_read_complete_fraghdr] rpc-service: Regular RPC header, remaining: 396
[2010-08-12 17:48:26.433050] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 396, dataread: 396
[2010-08-12 17:48:26.433065] T [rpcsvc.c:2333:rpcsvc_record_update_state] rpc-service: Regular frag
[2010-08-12 17:48:26.433080] T [rpcsvc.c:1158:rpcsvc_record_read_complete_frag] rpc-service: Fragment remaining: 0
[2010-08-12 17:48:26.433096] T [rpcsvc.c:2355:rpcsvc_record_update_state] rpc-service: Full Record Received.
[2010-08-12 17:48:26.433133] T [rpcsvc-auth.c:247:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_NULL
[2010-08-12 17:48:26.433148] D [rpcsvc.c:1817:rpcsvc_request_create] rpc-service: RPC XID: 3214aa67, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7
[2010-08-12 17:48:26.433164] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE
[2010-08-12 17:48:26.433183] D [nfs3-helpers.c:2348:nfs3_log_rw_call] nfs-nfsv3: XID: 3214aa67, WRITE: args: FH: hashcount 7, xlid 0, gen 5504105043429226257, ino 216894205, offset: 0,  count: 293, STABLE
[2010-08-12 17:48:26.433199] T [nfs3.c:1837:nfs3_write] nfs-nfsv3: FH to Volume: distribute
[2010-08-12 17:48:26.433216] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-12 17:48:26.433233] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-12 17:48:26.433266] T [nfs3-helpers.c:2192:nfs3_file_open_and_resume] nfs-nfsv3: Opening: /nfsalpha155/ip-10-244-167-207/test8/linux-2.6.35/Documentation/serial/.README.cycladesZ.XQLgvX
[2010-08-12 17:48:26.433284] T [nfs3-helpers.c:2175:nfs3_fdcache_getfd] nfs-nfsv3: fd not found in state
[2010-08-12 17:48:26.433303] T [nfs3-helpers.c:2050:__nfs3_get_inode_queue] nfs-nfsv3: Initing inode queue
[2010-08-12 17:48:26.433320] T [nfs3-helpers.c:2090:__nfs3_queue_call_state] nfs-nfsv3: First call in queue
[2010-08-12 17:48:26.433335] T [nfs3-helpers.c:2095:__nfs3_queue_call_state] nfs-nfsv3: Queueing call state
[2010-08-12 17:48:26.433349] T [nfs3-helpers.c:2152:__nfs3_file_open_and_resume] nfs-nfsv3: Opening uncached fd
[2010-08-12 17:48:26.433366] T [nfs-fops.c:1087:nfs_fop_open] nfs: Open: /nfsalpha155/ip-10-244-167-207/test8/linux-2.6.35/Documentation/serial/.README.cycladesZ.XQLgvX
[2010-08-12 17:48:26.433401] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.433418] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 260, payload: 132, rpc hdr: 128
[2010-08-12 17:48:26.433447] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.433463] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 260, payload: 132, rpc hdr: 128
[2010-08-12 17:48:26.433497] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.433517] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 260, payload: 132, rpc hdr: 128
[2010-08-12 17:48:26.453385] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 4, dataread: 4
[2010-08-12 17:48:26.453415] T [rpcsvc.c:1064:rpcsvc_record_read_complete_fraghdr] rpc-service: Received fragment size: 88
[2010-08-12 17:48:26.453432] T [rpcsvc.c:1071:rpcsvc_record_read_complete_fraghdr] rpc-service: Regular RPC header, remaining: 88
[2010-08-12 17:48:26.453456] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 88, dataread: 88
[2010-08-12 17:48:26.453472] T [rpcsvc.c:2333:rpcsvc_record_update_state] rpc-service: Regular frag
[2010-08-12 17:48:26.453491] T [rpcsvc.c:1158:rpcsvc_record_read_complete_frag] rpc-service: Fragment remaining: 0
[2010-08-12 17:48:26.453505] T [rpcsvc.c:2355:rpcsvc_record_update_state] rpc-service: Full Record Received.
[2010-08-12 17:48:26.453524] T [rpcsvc-auth.c:247:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_NULL
[2010-08-12 17:48:26.453539] D [rpcsvc.c:1817:rpcsvc_request_create] rpc-service: RPC XID: 3314aa67, Ver: 2, Program: 100003, ProgVers: 3, Proc: 6
[2010-08-12 17:48:26.453555] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - READ
[2010-08-12 17:48:26.453575] D [nfs3-helpers.c:2343:nfs3_log_rw_call] nfs-nfsv3: XID: 3314aa67, READ: args: FH: hashcount 4, xlid 0, gen 5504105043428906156, ino 69601214, offset: 70328320,  count: 4096
[2010-08-12 17:48:26.453593] T [nfs3.c:1514:nfs3_read] nfs-nfsv3: FH to Volume: distribute
[2010-08-12 17:48:26.453609] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-12 17:48:26.453627] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-12 17:48:26.453648] T [nfs3-helpers.c:2192:nfs3_file_open_and_resume] nfs-nfsv3: Opening: /nfsalpha152/ip-10-244-167-207/test15/fi3
[2010-08-12 17:48:26.453663] T [nfs3-helpers.c:2172:nfs3_fdcache_getfd] nfs-nfsv3: fd found in state: 2
[2010-08-12 17:48:26.453677] T [nfs3-helpers.c:1880:__nfs3_fdcache_update_entry] nfs-nfsv3: Updating fd: 0x2aaab2851150
[2010-08-12 17:48:26.453694] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-12 17:48:26.453729] D [client3_1-fops.c:3332:client3_1_readv] ec2-75-101-206-122.compute-1.amazonaws.com-3: (69601214): failed to get fd ctx. EBADFD
[2010-08-12 17:48:26.453749] D [client3_1-fops.c:3325:client3_1_readv] ec2-75-101-206-122.compute-1.amazonaws.com-1: (69601214): failed to get fd ctx. EBADFD
[2010-08-12 17:48:26.453788] T [socket.c:1689:socket_connect] ec2-75-101-206-122.compute-1.amazonaws.com-2: connect () called on transport already connected
[2010-08-12 17:48:26.453806] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.453824] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 172, payload: 44, rpc hdr: 128
[2010-08-12 17:48:26.453840] D [rpc-clnt.c:1287:rpc_clnt_submit] rpc-clnt: transmission of rpc-request failed
[2010-08-12 17:48:26.453864] D [nfs3-helpers.c:2382:nfs3_log_read_res] nfs-nfsv3: XID: 3314aa67, READ: NFS: 10006(Error occurred on the server or IO Error), POSIX: 107(Transport endpoint is not connected), count: 4294967295, is_eof: 0, vector: count: 1, len: 0
[2010-08-12 17:48:26.453885] T [rpcsvc.c:1572:rpcsvc_submit_generic] rpc-service: Tx message: 8
[2010-08-12 17:48:26.453902] T [rpcsvc.c:1365:rpcsvc_record_build_header] rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2010-08-12 17:48:26.453918] T [rpcsvc.c:1402:rpcsvc_conn_submit] rpc-service: Tx Header: 28, payload: 8
[2010-08-12 17:48:26.453938] T [nfs3.c:202:nfs3_call_state_wipe] nfs-nfsv3: fd ref: 3
[2010-08-12 17:48:26.453963] T [rpcsvc.c:2434:__rpcsvc_conn_data_poll_out] rpc-service: First Tx Buf
[2010-08-12 17:48:26.453990] T [rpcsvc.c:2445:__rpcsvc_conn_data_poll_out] rpc-service: conn: 0x2aaab4209dc0, Tx request: 28, Tx sent: 28
[2010-08-12 17:48:26.454008] T [rpcsvc.c:2441:__rpcsvc_conn_data_poll_out] rpc-service: Last Tx Buf
[2010-08-12 17:48:26.454029] T [rpcsvc.c:2445:__rpcsvc_conn_data_poll_out] rpc-service: conn: 0x2aaab4209dc0, Tx request: 8, Tx sent: 8
[2010-08-12 17:48:26.454371] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 4, dataread: 4
[2010-08-12 17:48:26.454388] T [rpcsvc.c:1064:rpcsvc_record_read_complete_fraghdr] rpc-service: Received fragment size: 88
[2010-08-12 17:48:26.454403] T [rpcsvc.c:1071:rpcsvc_record_read_complete_fraghdr] rpc-service: Regular RPC header, remaining: 88
[2010-08-12 17:48:26.454421] T [rpcsvc.c:2396:rpcsvc_conn_data_poll_in] rpc-service: conn: 0x2aaab4209dc0, readsize: 88, dataread: 88
[2010-08-12 17:48:26.454436] T [rpcsvc.c:2333:rpcsvc_record_update_state] rpc-service: Regular frag
[2010-08-12 17:48:26.454450] T [rpcsvc.c:1158:rpcsvc_record_read_complete_frag] rpc-service: Fragment remaining: 0
[2010-08-12 17:48:26.454465] T [rpcsvc.c:2355:rpcsvc_record_update_state] rpc-service: Full Record Received.
[2010-08-12 17:48:26.454479] T [rpcsvc-auth.c:247:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_NULL
[2010-08-12 17:48:26.454494] D [rpcsvc.c:1817:rpcsvc_request_create] rpc-service: RPC XID: 3414aa67, Ver: 2, Program: 100003, ProgVers: 3, Proc: 6
[2010-08-12 17:48:26.454509] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - READ
[2010-08-12 17:48:26.454526] D [nfs3-helpers.c:2343:nfs3_log_rw_call] nfs-nfsv3: XID: 3414aa67, READ: args: FH: hashcount 4, xlid 0, gen 5504105043428906503, ino 656868096, offset: 12808192,  count: 4096
[2010-08-12 17:48:26.454542] T [nfs3.c:1514:nfs3_read] nfs-nfsv3: FH to Volume: distribute
[2010-08-12 17:48:26.454556] T [nfs3-helpers.c:2973:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-08-12 17:48:26.454576] T [nfs3-helpers.c:2477:nfs3_fh_resolve_inode_done] nfs-nfsv3: FH inode resolved
[2010-08-12 17:48:26.454594] T [nfs3-helpers.c:2192:nfs3_file_open_and_resume] nfs-nfsv3: Opening: /nfsalpha151/ip-10-244-167-207/test15/fi2
[2010-08-12 17:48:26.454623] T [nfs3-helpers.c:2172:nfs3_fdcache_getfd] nfs-nfsv3: fd found in state: 2
[2010-08-12 17:48:26.454637] T [nfs3-helpers.c:1880:__nfs3_fdcache_update_entry] nfs-nfsv3: Updating fd: 0x2aaab720b260
[2010-08-12 17:48:26.454652] T [nfs.c:409:nfs_user_create] nfs: uid: 0, gid 0, gids: 0
[2010-08-12 17:48:26.454673] D [client3_1-fops.c:3332:client3_1_readv] ec2-174-129-137-26.compute-1.amazonaws.com-2: (656868096): failed to get fd ctx. EBADFD
[2010-08-12 17:48:26.454693] T [rpc-clnt.c:1164:rpc_clnt_record] : Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2010-08-12 17:48:26.454729] T [rpc-clnt.c:1056:rpc_clnt_record_build_header] rpc-clnt: Request fraglen 172, payload: 44, rpc hdr: 128
/lib64/libc.so.6[0x2aaaab7aaf30]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/nfs/server.so[0x2aaaaca1b33f]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/nfs/server.so[0x2aaaaca0f491]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/nfs/server.so[0x2aaaaca0fd38]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/nfs/server.so[0x2aaaaca1f6c7]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/nfs/server.so[0x2aaaac9fd145]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/cluster/distribute.so(dht_lookup_cbk+0x871)[0x2aaaac7d488a]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/cluster/replicate.so[0x2aaaac568dd8]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/cluster/replicate.so(afr_fresh_lookup_cbk+0x535)[0x2aaaac569379]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/xlator/protocol/client.so(client3_1_lookup_cbk+0x727)[0x2aaaac34b6fa]
/opt/glusterfs/3.1.0qa3//lib/libgfrpc.so.0(saved_frames_unwind+0x273)[0x2aaaaaf32654]
/opt/glusterfs/3.1.0qa3//lib/libgfrpc.so.0(saved_frames_destroy+0x15)[0x2aaaaaf326de]
/opt/glusterfs/3.1.0qa3//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xee)[0x2aaaaaf32ada]
/opt/glusterfs/3.1.0qa3//lib/libgfrpc.so.0(rpc_clnt_notify+0x14e)[0x2aaaaaf33233]
/opt/glusterfs/3.1.0qa3//lib/libgfrpc.so.0(rpc_transport_notify+0xd8)[0x2aaaaaf3109b]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/rpc-transport/socket.so(socket_event_poll_err+0x70)[0x2aaaacef6c09]
/opt/glusterfs/3.1.0qa3//lib/glusterfs/3.1.0qa3/rpc-transport/socket.so(socket_event_handler+0x10e)[0x2aaaacef9b0e]
/opt/glusterfs/3.1.0qa3//lib/libglusterfs.so.0[0x2aaaaad038e4]
/opt/glusterfs/3.1.0qa3//lib/libglusterfs.so.0[0x2aaaaad03ac5]
/opt/glusterfs/3.1.0qa3//lib/libglusterfs.so.0(event_dispatch+0x73)[0x2aaaaad03ddb]
/opt/glusterfs/3.1.0qa3/sbin/glusterfsd(main+0xec)[0x405062]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2aaaab798074]
/opt/glusterfs/3.1.0qa3/sbin/glusterfsd[0x4027a9]
---------
#0  0x00002aaaaca1b33f in nfs3_fill_create3res (res=0x7fff2a33cfa0, stat=NFS3_OK, newfh=0x0, newbuf=0x0, preparent=0x0, postparent=0x0) at nfs3-helpers.c:1223
1223	        nfs3_map_xlid_to_statdev (newbuf, newfh->xlatorid);
(gdb) bt full
#0  0x00002aaaaca1b33f in nfs3_fill_create3res (res=0x7fff2a33cfa0, stat=NFS3_OK, newfh=0x0, newbuf=0x0, preparent=0x0, postparent=0x0) at nfs3-helpers.c:1223
	poa = {attributes_follow = 0, post_op_attr_u = {attributes = {type = 0, mode = 0, nlink = 0, uid = 0, gid = 0, size = 0, used = 0, rdev = {specdata1 = 0, 
        specdata2 = 0}, fsid = 0, fileid = 0, atime = {seconds = 0, nseconds = 0}, mtime = {seconds = 0, nseconds = 0}, ctime = {seconds = 0, nseconds = 0}}}}
	dirwcc = {before = {attributes_follow = 0, pre_op_attr_u = {attributes = {size = 0, mtime = {seconds = 0, nseconds = 0}, ctime = {seconds = 0, 
          nseconds = 0}}}}, after = {attributes_follow = 0, post_op_attr_u = {attributes = {type = 0, mode = 0, nlink = 0, uid = 0, gid = 0, size = 0, used = 0, 
        rdev = {specdata1 = 0, specdata2 = 0}, fsid = 0, fileid = 0, atime = {seconds = 0, nseconds = 0}, mtime = {seconds = 0, nseconds = 0}, ctime = {seconds = 0, 
          nseconds = 0}}}}}
#1  0x00002aaaaca0f491 in nfs3_create_reply (req=0x2aaab6763530, stat=NFS3_OK, newfh=0x0, newbuf=0x0, preparent=0x0, postparent=0x0) at nfs3.c:1991
	res = {status = NFS3_OK, create3res_u = {resok = {obj = {handle_follows = 0, post_op_fh3_u = {handle = {data = {data_len = 0, data_val = 0x0}}}}, 
      obj_attributes = {attributes_follow = 0, post_op_attr_u = {attributes = {type = 0, mode = 0, nlink = 0, uid = 0, gid = 0, size = 0, used = 0, rdev = {
              specdata1 = 0, specdata2 = 0}, fsid = 0, fileid = 0, atime = {seconds = 0, nseconds = 0}, mtime = {seconds = 0, nseconds = 0}, ctime = {seconds = 0, 
              nseconds = 0}}}}, dir_wcc = {before = {attributes_follow = 0, pre_op_attr_u = {attributes = {size = 0, mtime = {seconds = 0, nseconds = 0}, ctime = {
                seconds = 0, nseconds = 0}}}}, after = {attributes_follow = 0, post_op_attr_u = {attributes = {type = 0, mode = 0, nlink = 0, uid = 0, gid = 0, 
              size = 0, used = 0, rdev = {specdata1 = 0, specdata2 = 0}, fsid = 0, fileid = 0, atime = {seconds = 0, nseconds = 0}, mtime = {seconds = 0, 
                nseconds = 0}, ctime = {seconds = 0, nseconds = 0}}}}}}, resfail = {dir_wcc = {before = {attributes_follow = 0, pre_op_attr_u = {attributes = {
              size = 0, mtime = {seconds = 0, nseconds = 0}, ctime = {seconds = 0, nseconds = 0}}}}, after = {attributes_follow = 0, post_op_attr_u = {attributes = {
              type = 0, mode = 0, nlink = 0, uid = 0, gid = 0, size = 0, used = 0, rdev = {specdata1 = 0, specdata2 = 0}, fsid = 0, fileid = 0, atime = {
                seconds = 0, nseconds = 0}, mtime = {seconds = 0, nseconds = 0}, ctime = {seconds = 0, nseconds = 0}}}}}}}}
#2  0x00002aaaaca0fd38 in nfs3_create_resume (carg=0x2aaaaf4fcaa0) at nfs3.c:2201
	stat = NFS3_OK
	ret = -14
	cs = (nfs3_call_state_t *) 0x2aaaaf4fcaa0
#3  0x00002aaaaca1f6c7 in nfs3_fh_resolve_entry_lookup_cbk (frame=0x7bafed8, cookie=0x63df70, this=0x63cf40, op_ret=-1, op_errno=0, inode=0x0, buf=0x7beca88, 
    xattr=0x0, postparent=0x7becb58) at nfs3-helpers.c:2560
	cs = (nfs3_call_state_t *) 0x2aaaaf4fcaa0
	__FUNCTION__ = "nfs3_fh_resolve_entry_lookup_cbk"
#4  0x00002aaaac9fd145 in nfs_fop_lookup_cbk (frame=0x7bafed8, cookie=0x63df70, this=0x63cf40, op_ret=-1, op_errno=0, inode=0x0, buf=0x7beca88, xattr=0x0, 
    postparent=0x7becb58) at nfs-fops.c:260
	local = (struct nfs_fop_local *) 0x2aaaaab38e60
	progcbk = (fop_lookup_cbk_t) 0x2aaaaca1f57d <nfs3_fh_resolve_entry_lookup_cbk>
#5  0x00002aaaac7d488a in dht_lookup_cbk (frame=0x7baff60, cookie=0x7bafd20, this=0x63cf40, op_ret=-1, op_errno=0, inode=0x0, stbuf=0x7beca88, xattr=0x0, 
    postparent=0x7becb58) at dht-common.c:776
	fn = (fop_lookup_cbk_t) 0x2aaaac9fd041 <nfs_fop_lookup_cbk>
	_parent = (call_frame_t *) 0x7bafed8
	old_THIS = (xlator_t *) 0x63cf40
	__local = (dht_local_t *) 0x7bc9710
	__xl = (xlator_t *) 0x63cf40
Missing separate debuginfos, use: debuginfo-install gcc.x86_64 glibc.x86_64
---Type <return> to continue, or q <return> to quit--- 
	is_linkfile = 0 '\0'
	is_dir = 0 '\0'
	subvol = (xlator_t *) 0x0
	conf = (dht_conf_t *) 0x63f7d0
	local = (dht_local_t *) 0x7bc9710
	loc = (loc_t *) 0x7bc9718
	prev = (call_frame_t *) 0x7bafd20
	ret = 0
	tmp_layout = 0
	parent_layout = (dht_layout_t *) 0x0
	__FUNCTION__ = "dht_lookup_cbk"
#6  0x00002aaaac568dd8 in afr_lookup_done (frame=0x7bafd20, this=0x63b940, lookup_buf=0x7beca88) at afr.c:695
	fn = (fop_lookup_cbk_t) 0x2aaaac7d4019 <dht_lookup_cbk>
	_parent = (call_frame_t *) 0x7baff60
	old_THIS = (xlator_t *) 0x63b940
	__local = (afr_local_t *) 0x7bec940
	__this = (xlator_t *) 0x63b940
	unwind = 1
	source = -1
	sh_type_str = '\0' <repeats 255 times>
	local = (afr_local_t *) 0x7bec940
	__FUNCTION__ = "afr_lookup_done"
#7  0x00002aaaac569379 in afr_fresh_lookup_cbk (frame=0x7bafd20, cookie=0x1, this=0x63b940, op_ret=-1, op_errno=0, inode=0x7bafdc0, buf=0x7fff2a33d640, xattr=0x0, 
    postparent=0x7fff2a33d5d0) at afr.c:842
	local = (afr_local_t *) 0x7bec940
	priv = (afr_private_t *) 0x63fb40
	lookup_buf = (struct iatt *) 0x7beca88
	call_count = 0
	child_index = 1
	first_up_child = -1
#8  0x00002aaaac34b6fa in client3_1_lookup_cbk (req=0x2aaaad732da0, iov=0x7fff2a33d840, count=1, myframe=0x7be0f70) at client3_1-fops.c:1955
	fn = (fop_lookup_cbk_t) 0x2aaaac568e44 <afr_fresh_lookup_cbk>
	_parent = (call_frame_t *) 0x7bafd20
	old_THIS = (xlator_t *) 0x632360
	local = (clnt_local_t *) 0x7be0ff0
	frame = (call_frame_t *) 0x7be0f70
	ret = 0
---Type <return> to continue, or q <return> to quit---
	rsp = {gfs_id = 0, op_ret = -1, op_errno = 0, stat = {ia_ino = 0, ia_gen = 0, ia_dev = 0, mode = 0, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_rdev = 0, 
    ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0}, postparent = {
    ia_ino = 0, ia_gen = 0, ia_dev = 0, mode = 0, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, 
    ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0}, dict = {dict_len = 0, dict_val = 0x0}}
	stbuf = {ia_ino = 0, ia_gen = 0, ia_dev = 0, ia_type = IA_INVAL, ia_prot = {suid = 0 '\0', sgid = 0 '\0', sticky = 0 '\0', owner = {read = 0 '\0', 
      write = 0 '\0', exec = 0 '\0'}, group = {read = 0 '\0', write = 0 '\0', exec = 0 '\0'}, other = {read = 0 '\0', write = 0 '\0', exec = 0 '\0'}}, ia_nlink = 0, 
  ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, 
  ia_ctime_nsec = 0}
	postparent = {ia_ino = 0, ia_gen = 0, ia_dev = 0, ia_type = IA_INVAL, ia_prot = {suid = 0 '\0', sgid = 0 '\0', sticky = 0 '\0', owner = {read = 0 '\0', 
      write = 0 '\0', exec = 0 '\0'}, group = {read = 0 '\0', write = 0 '\0', exec = 0 '\0'}, other = {read = 0 '\0', write = 0 '\0', exec = 0 '\0'}}, ia_nlink = 0, 
  ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, 
  ia_ctime_nsec = 0}
	op_errno = 0
	oldino = 0
	oldgen = 0
	xattr = (dict_t *) 0x0
	inode = (inode_t *) 0x7bafdc0
	buf = 0x0
	__FUNCTION__ = "client3_1_lookup_cbk"
#9  0x00002aaaaaf32654 in saved_frames_unwind (saved_frames=0x6433c0) at rpc-clnt.c:335
	trav = (struct saved_frame *) 0x2aaaad973560
	tmp = (struct saved_frame *) 0x2aaaad9734f0
	frame_sent_tm = (struct tm *) 0x2aaaabacfce0
	timestr = "2010-08-12 17:48:26.117373", '\0' <repeats 229 times>
	iov = {iov_base = 0x0, iov_len = 0}
	__FUNCTION__ = "saved_frames_unwind"
#10 0x00002aaaaaf326de in saved_frames_destroy (frames=0x6433c0) at rpc-clnt.c:348
No locals.
#11 0x00002aaaaaf32ada in rpc_clnt_connection_cleanup (conn=0x642c80) at rpc-clnt.c:472
	saved_frames = (struct saved_frames *) 0x6433c0
	clnt = (struct rpc_clnt *) 0x642c50
	__FUNCTION__ = "rpc_clnt_connection_cleanup"
#12 0x00002aaaaaf33233 in rpc_clnt_notify (trans=0x642e10, mydata=0x642c80, event=RPC_TRANSPORT_DISCONNECT, data=0x642e10) at rpc-clnt.c:743
	conn = (rpc_clnt_connection_t *) 0x642c80
	clnt = (struct rpc_clnt *) 0x642c50
	ret = -1
	req_info = (rpc_request_info_t *) 0x0
---Type <return> to continue, or q <return> to quit---
	pollin = (rpc_transport_pollin_t *) 0x0
	tv = {tv_sec = 0, tv_usec = 0}
#13 0x00002aaaaaf3109b in rpc_transport_notify (this=0x642e10, event=RPC_TRANSPORT_DISCONNECT, data=0x642e10) at rpc-transport.c:1239
	ret = -1
#14 0x00002aaaacef6c09 in socket_event_poll_err (this=0x642e10) at socket.c:531
	priv = (socket_private_t *) 0x6432a0
	ret = -1
#15 0x00002aaaacef9b0e in socket_event_handler (fd=8, idx=1, data=0x642e10, poll_in=1, poll_out=0, poll_err=24) at socket.c:1517
	this = (rpc_transport_t *) 0x642e10
	priv = (socket_private_t *) 0x6432a0
	ret = -1
	__FUNCTION__ = "socket_event_handler"
#16 0x00002aaaaad038e4 in event_dispatch_epoll_handler (event_pool=0x629330, events=0x6489c0, i=0) at event.c:812
	event_data = (struct event_data *) 0x6489c4
	handler = (event_handler_t) 0x2aaaacef9a00 <socket_event_handler>
	data = (void *) 0x642e10
	idx = 1
	ret = -1
	__FUNCTION__ = "event_dispatch_epoll_handler"
#17 0x00002aaaaad03ac5 in event_dispatch_epoll (event_pool=0x629330) at event.c:876
	events = (struct epoll_event *) 0x6489c0
	size = 1
	i = 0
	ret = 1
	__FUNCTION__ = "event_dispatch_epoll"
#18 0x00002aaaaad03ddb in event_dispatch (event_pool=0x629330) at event.c:984
	ret = -1
	__FUNCTION__ = "event_dispatch"
#19 0x0000000000405062 in main (argc=7, argv=0x7fff2a33de58) at glusterfsd.c:1273
	ctx = (glusterfs_ctx_t *) 0x629010
	ret = 0