Attachment 583811 Details for Bug 820887 – Mount log file

Mount log file

mnt-gfsc1.log (text/x-log), 92.29 KB, created by Shwetha Panduranga on 2012-05-11 11:37:18 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Shwetha Panduranga

Created: 2012-05-11 11:37:18 UTC

Size: 92.29 KB

patch

obsolete

>[2012-05-11 05:18:03.231152] I [glusterfsd.c:1629:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.3.0qa40
>[2012-05-11 05:18:04.397683] I [io-cache.c:1558:check_cache_size_ok] 0-vol-quick-read: Max cache size is 2104397824
>[2012-05-11 05:18:04.405687] I [io-cache.c:1558:check_cache_size_ok] 0-vol-io-cache: Max cache size is 2104397824
>[2012-05-11 05:18:04.513474] I [client.c:2151:notify] 0-vol-client-0: parent translators are ready, attempting connect on transport
>[2012-05-11 05:18:04.610332] I [client.c:2151:notify] 0-vol-client-1: parent translators are ready, attempting connect on transport
>[2012-05-11 05:18:04.703771] I [client.c:2151:notify] 0-vol-client-2: parent translators are ready, attempting connect on transport
>[2012-05-11 05:18:04.797745] I [client.c:2151:notify] 0-vol-client-3: parent translators are ready, attempting connect on transport
>[2012-05-11 05:18:04.891178] I [client.c:2151:notify] 0-vol-client-4: parent translators are ready, attempting connect on transport
>[2012-05-11 05:18:04.985268] I [client.c:2151:notify] 0-vol-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume vol-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.58
>  4:     option remote-subvolume /export/brick0
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume vol-client-1
>  9:     type protocol/client
> 10:     option remote-host 172.17.251.59
> 11:     option remote-subvolume /export/brick1
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume vol-client-2
> 16:     type protocol/client
> 17:     option remote-host 172.17.251.60
> 18:     option remote-subvolume /export/brick2
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume vol-client-3
> 23:     type protocol/client
> 24:     option remote-host 172.17.251.58
> 25:     option remote-subvolume /export/brick3
> 26:     option transport-type tcp
> 27: end-volume
> 28: 
> 29: volume vol-client-4
> 30:     type protocol/client
> 31:     option remote-host 172.17.251.59
> 32:     option remote-subvolume /export/brick4
> 33:     option transport-type tcp
> 34: end-volume
> 35: 
> 36: volume vol-client-5
> 37:     type protocol/client
> 38:     option remote-host 172.17.251.60
> 39:     option remote-subvolume /export/brick5
> 40:     option transport-type tcp
> 41: end-volume
> 42: 
> 43: volume vol-replicate-0
> 44:     type cluster/replicate
> 45:     subvolumes vol-client-0 vol-client-1 vol-client-2
> 46: end-volume
> 47: 
> 48: volume vol-replicate-1
> 49:     type cluster/replicate
> 50:     subvolumes vol-client-3 vol-client-4 vol-client-5
> 51: end-volume
> 52: 
> 53: volume vol-dht
> 54:     type cluster/distribute
> 55:     subvolumes vol-replicate-0 vol-replicate-1
> 56: end-volume
> 57: 
> 58: volume vol-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes vol-dht
> 61: end-volume
> 62: 
> 63: volume vol-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes vol-write-behind
> 66: end-volume
> 67: 
> 68: volume vol-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes vol-read-ahead
> 71: end-volume
> 72: 
> 73: volume vol-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes vol-io-cache
> 76: end-volume
> 77: 
> 78: volume vol-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes vol-quick-read
> 81: end-volume
> 82: 
> 83: volume vol
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes vol-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-05-11 05:18:05.127991] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-0: changing port to 24009 (from 0)
>[2012-05-11 05:18:05.132838] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-1: changing port to 24009 (from 0)
>[2012-05-11 05:18:05.133276] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-2: changing port to 24009 (from 0)
>[2012-05-11 05:18:05.133787] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-3: changing port to 24010 (from 0)
>[2012-05-11 05:18:05.150178] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-4: changing port to 24010 (from 0)
>[2012-05-11 05:18:05.150850] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-5: changing port to 24010 (from 0)
>[2012-05-11 05:18:08.974565] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:18:09.004539] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'.
>[2012-05-11 05:18:09.010041] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:18:09.023797] I [afr-common.c:3629:afr_notify] 0-vol-replicate-0: Subvolume 'vol-client-0' came back up; going online.
>[2012-05-11 05:18:09.035535] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-0: Server lk version = 1
>[2012-05-11 05:18:09.116149] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-1: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:18:09.121610] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-1: Connected to 172.17.251.59:24009, attached to remote volume '/export/brick1'.
>[2012-05-11 05:18:09.122014] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:18:09.124502] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-1: Server lk version = 1
>[2012-05-11 05:18:09.318805] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-2: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:18:09.325250] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-2: Connected to 172.17.251.60:24009, attached to remote volume '/export/brick2'.
>[2012-05-11 05:18:09.325688] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:18:09.376796] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-2: Server lk version = 1
>[2012-05-11 05:18:09.608519] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:18:09.613876] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'.
>[2012-05-11 05:18:09.614098] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:18:09.614773] I [afr-common.c:3629:afr_notify] 0-vol-replicate-1: Subvolume 'vol-client-3' came back up; going online.
>[2012-05-11 05:18:09.615277] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-3: Server lk version = 1
>[2012-05-11 05:18:09.711883] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-4: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:18:09.713446] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-4: Connected to 172.17.251.59:24010, attached to remote volume '/export/brick4'.
>[2012-05-11 05:18:09.713672] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:18:09.714592] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-4: Server lk version = 1
>[2012-05-11 05:18:09.814541] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-5: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:18:09.816277] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-5: Connected to 172.17.251.60:24010, attached to remote volume '/export/brick5'.
>[2012-05-11 05:18:09.816518] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:18:09.885727] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-05-11 05:18:09.890548] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-5: Server lk version = 1
>[2012-05-11 05:18:09.903244] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-05-11 05:18:10.004028] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 0-vol-replicate-0: added root inode
>[2012-05-11 05:18:10.122141] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 0-vol-replicate-1: added root inode
>[2012-05-11 05:30:17.603601] W [socket.c:1521:__socket_proto_state_machine] 0-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009)
>[2012-05-11 05:30:17.609434] I [client.c:2099:client_rpc_notify] 0-vol-client-0: disconnected
>[2012-05-11 05:30:22.223944] W [socket.c:204:__socket_rwv] 0-vol-client-3: writev failed (Broken pipe)
>[2012-05-11 05:30:22.267448] I [socket.c:1807:socket_event_handler] 0-transport: disconnecting now
>[2012-05-11 05:30:22.402461] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.224378 (xid=0x92785x)
>[2012-05-11 05:30:22.419607] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.422332] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.231646 (xid=0x92786x)
>[2012-05-11 05:30:22.422708] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.423246] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.237831 (xid=0x92787x)
>[2012-05-11 05:30:22.423590] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.429332] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.238971 (xid=0x92788x)
>[2012-05-11 05:30:22.430776] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.432882] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.242193 (xid=0x92789x)
>[2012-05-11 05:30:22.433237] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.433803] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.243210 (xid=0x92790x)
>[2012-05-11 05:30:22.434124] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.434667] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.249772 (xid=0x92791x)
>[2012-05-11 05:30:22.434977] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.440550] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.254993 (xid=0x92792x)
>[2012-05-11 05:30:22.441894] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.443442] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.256066 (xid=0x92793x)
>[2012-05-11 05:30:22.443786] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.444316] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:22.203825 (xid=0x92783x)
>[2012-05-11 05:30:22.445234] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.445869] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:22.214829 (xid=0x92784x)
>[2012-05-11 05:30:22.446183] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.451019] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:22.261213 (xid=0x92794x)
>[2012-05-11 05:30:22.455535] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:22.459044] I [client.c:2099:client_rpc_notify] 0-vol-client-3: disconnected
>[2012-05-11 05:30:22.463296] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.476787] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.478253] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.483383] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.487423] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.494090] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.495557] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.501414] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.530972] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.544974] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.553846] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.556724] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.557697] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.558897] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.566596] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.570265] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.571465] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:22.578989] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:28.277359] E [socket.c:1724:socket_connect_finish] 0-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused)
>[2012-05-11 05:30:33.124536] E [socket.c:1724:socket_connect_finish] 0-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused)
>[2012-05-11 05:30:52.747429] W [socket.c:204:__socket_rwv] 0-vol-client-4: writev failed (Connection reset by peer)
>[2012-05-11 05:30:52.752393] I [socket.c:1807:socket_event_handler] 0-transport: disconnecting now
>[2012-05-11 05:30:52.753185] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.725643 (xid=0x98227x)
>[2012-05-11 05:30:52.757681] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:52.758367] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.728072 (xid=0x98228x)
>[2012-05-11 05:30:52.758737] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:52.759280] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.732929 (xid=0x98229x)
>[2012-05-11 05:30:52.764772] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:52.765438] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.742158 (xid=0x98230x)
>[2012-05-11 05:30:52.765772] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:52.766324] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.747726 (xid=0x98231x)
>[2012-05-11 05:30:52.770810] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:30:52.771111] I [client.c:2099:client_rpc_notify] 0-vol-client-4: disconnected
>[2012-05-11 05:30:52.791233] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.798062] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.804841] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.830089] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.837645] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.844871] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.845995] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.853042] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.854133] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.859431] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.867732] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.869020] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:52.907666] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4:  (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD
>[2012-05-11 05:30:55.113321] W [socket.c:1521:__socket_proto_state_machine] 0-vol-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.59:24009)
>[2012-05-11 05:30:55.113815] I [client.c:2099:client_rpc_notify] 0-vol-client-1: disconnected
>[2012-05-11 05:31:03.975636] E [socket.c:1724:socket_connect_finish] 0-vol-client-4: connection to 172.17.251.59:24010 failed (Connection refused)
>[2012-05-11 05:31:05.585948] E [socket.c:1724:socket_connect_finish] 0-vol-client-1: connection to 172.17.251.59:24009 failed (Connection refused)
>[2012-05-11 05:34:30.748825] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:34:30.755305] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'.
>[2012-05-11 05:34:30.757686] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:34:30.759409] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-0: Server lk version = 1
>[2012-05-11 05:34:32.129841] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-4: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:34:32.137772] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-4: Connected to 172.17.251.59:24010, attached to remote volume '/export/brick4'.
>[2012-05-11 05:34:32.138094] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:34:32.144126] I [client-handshake.c:1283:client_post_handshake] 0-vol-client-4: 1 fds open - Delaying child_up until they are re-opened
>[2012-05-11 05:34:32.206814] I [client-lk.c:610:decrement_reopen_fd_count] 0-vol-client-4: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-05-11 05:34:32.233428] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-4: Server lk version = 1
>[2012-05-11 05:34:32.302017] I [afr-common.c:1347:afr_launch_self_heal] 0-vol-replicate-1: background  data missing-entry gfid self-heal triggered. path: /fuse1.1/dir.1/file.3, reason: subvolume came online
>[2012-05-11 05:34:32.379047] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc
>[2012-05-11 05:34:32.414802] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.425423] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.425937] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.426360] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.426785] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.427181] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.427620] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.428027] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.464050] W [client3_1-fops.c:419:client3_1_open_cbk] 0-vol-client-4: remote operation failed: No such file or directory. Path: /fuse1.1/dir.1/file.3
>[2012-05-11 05:34:32.593230] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-vol-replicate-1: path /fuse1.1/dir.1/file.3 on subvolume vol-client-4 => -1 (No such file or directory)
>[2012-05-11 05:34:32.782404] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc
>[2012-05-11 05:34:32.783182] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.783664] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.784073] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.784522] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.784942] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.788684] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.789166] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.789607] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:32.798574] I [afr-open.c:327:afr_openfd_fix_open_cbk] 0-vol-replicate-1: fd for /fuse1.1/dir.1/file.3 opened successfully on subvolume vol-client-4
>[2012-05-11 05:34:33.623453] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:34:33.689286] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'.
>[2012-05-11 05:34:33.689650] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:34:33.689848] I [client-handshake.c:1283:client_post_handshake] 0-vol-client-3: 1 fds open - Delaying child_up until they are re-opened
>[2012-05-11 05:34:33.703704] I [client-lk.c:610:decrement_reopen_fd_count] 0-vol-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-05-11 05:34:33.740093] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-3: Server lk version = 1
>[2012-05-11 05:34:33.998364] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-1: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:34:34.001735] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-1: Connected to 172.17.251.59:24009, attached to remote volume '/export/brick1'.
>[2012-05-11 05:34:34.001968] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:34:34.011245] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-1: Server lk version = 1
>[2012-05-11 05:34:34.018863] I [afr-common.c:1347:afr_launch_self_heal] 0-vol-replicate-1: background  data missing-entry gfid self-heal triggered. path: /fuse1.1/dir.1/file.3, reason: subvolume came online
>[2012-05-11 05:34:34.020187] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc
>[2012-05-11 05:34:34.020994] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.021469] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.021899] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.022305] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.022743] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.023160] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.031623] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.032140] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.035065] W [client3_1-fops.c:419:client3_1_open_cbk] 0-vol-client-3: remote operation failed: No such file or directory. Path: /fuse1.1/dir.1/file.3
>[2012-05-11 05:34:34.116324] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-vol-replicate-1: path /fuse1.1/dir.1/file.3 on subvolume vol-client-3 => -1 (No such file or directory)
>[2012-05-11 05:34:34.169808] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc
>[2012-05-11 05:34:34.170677] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.171140] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.171584] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.172003] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.175680] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.176154] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.176603] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.177020] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.187130] W [client3_1-fops.c:419:client3_1_open_cbk] 0-vol-client-3: remote operation failed: No such file or directory. Path: /fuse1.1/dir.1/file.3
>[2012-05-11 05:34:34.326411] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc
>[2012-05-11 05:34:34.327192] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.327679] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.328084] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.328505] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.328930] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.336358] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.336988] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.337420] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:34:34.339415] I [afr-open.c:327:afr_openfd_fix_open_cbk] 0-vol-replicate-1: fd for /fuse1.1/dir.1/file.3 opened successfully on subvolume vol-client-3
>[2012-05-11 05:35:02.260508] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-05-11 05:35:03.629567] I [io-cache.c:1558:check_cache_size_ok] 1-vol-quick-read: Max cache size is 2104397824
>[2012-05-11 05:35:03.630562] I [io-cache.c:1558:check_cache_size_ok] 1-vol-io-cache: Max cache size is 2104397824
>[2012-05-11 05:35:03.640219] I [quota.c:3085:quota_parse_limits] 1-vol-quota: no "limit-set" option provided
>[2012-05-11 05:36:42.199386] I [client.c:2151:notify] 1-vol-client-0: parent translators are ready, attempting connect on transport
>[2012-05-11 05:36:42.299897] I [client.c:2151:notify] 1-vol-client-1: parent translators are ready, attempting connect on transport
>[2012-05-11 05:36:42.391690] I [client.c:2151:notify] 1-vol-client-2: parent translators are ready, attempting connect on transport
>[2012-05-11 05:36:42.483385] I [client.c:2151:notify] 1-vol-client-3: parent translators are ready, attempting connect on transport
>[2012-05-11 05:36:42.575198] I [client.c:2151:notify] 1-vol-client-4: parent translators are ready, attempting connect on transport
>[2012-05-11 05:36:42.667066] I [client.c:2151:notify] 1-vol-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume vol-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.58
>  4:     option remote-subvolume /export/brick0
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume vol-client-1
>  9:     type protocol/client
> 10:     option remote-host 172.17.251.59
> 11:     option remote-subvolume /export/brick1
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume vol-client-2
> 16:     type protocol/client
> 17:     option remote-host 172.17.251.60
> 18:     option remote-subvolume /export/brick2
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume vol-client-3
> 23:     type protocol/client
> 24:     option remote-host 172.17.251.58
> 25:     option remote-subvolume /export/brick3
> 26:     option transport-type tcp
> 27: end-volume
> 28: 
> 29: volume vol-client-4
> 30:     type protocol/client
> 31:     option remote-host 172.17.251.59
> 32:     option remote-subvolume /export/brick4
> 33:     option transport-type tcp
> 34: end-volume
> 35: 
> 36: volume vol-client-5
> 37:     type protocol/client
> 38:     option remote-host 172.17.251.60
> 39:     option remote-subvolume /export/brick5
> 40:     option transport-type tcp
> 41: end-volume
> 42: 
> 43: volume vol-replicate-0
> 44:     type cluster/replicate
> 45:     subvolumes vol-client-0 vol-client-1 vol-client-2
> 46: end-volume
> 47: 
> 48: volume vol-replicate-1
> 49:     type cluster/replicate
> 50:     subvolumes vol-client-3 vol-client-4 vol-client-5
> 51: end-volume
> 52: 
> 53: volume vol-dht
> 54:     type cluster/distribute
> 55:     subvolumes vol-replicate-0 vol-replicate-1
> 56: end-volume
> 57: 
> 58: volume vol-quota
> 59:     type features/quota
> 60:     option timeout 0
> 61:     subvolumes vol-dht
> 62: end-volume
> 63: 
> 64: volume vol-write-behind
> 65:     type performance/write-behind
> 66:     subvolumes vol-quota
> 67: end-volume
> 68: 
> 69: volume vol-read-ahead
> 70:     type performance/read-ahead
> 71:     subvolumes vol-write-behind
> 72: end-volume
> 73: 
> 74: volume vol-io-cache
> 75:     type performance/io-cache
> 76:     subvolumes vol-read-ahead
> 77: end-volume
> 78: 
> 79: volume vol-quick-read
> 80:     type performance/quick-read
> 81:     subvolumes vol-io-cache
> 82: end-volume
> 83: 
> 84: volume vol-md-cache
> 85:     type performance/md-cache
> 86:     subvolumes vol-quick-read
> 87: end-volume
> 88: 
> 89: volume vol
> 90:     type debug/io-stats
> 91:     option latency-measurement off
> 92:     option count-fop-hits off
> 93:     subvolumes vol-md-cache
> 94: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-05-11 05:36:42.763494] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-05-11 05:36:42.770204] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-1: changing port to 24009 (from 0)
>[2012-05-11 05:36:42.770732] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-2: changing port to 24009 (from 0)
>[2012-05-11 05:36:42.771168] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-4: changing port to 24010 (from 0)
>[2012-05-11 05:36:42.772664] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-5: changing port to 24010 (from 0)
>[2012-05-11 05:36:42.841122] I [quota.c:3060:quota_parse_limits] 1-vol-quota: /:161061273600
>[2012-05-11 05:36:42.842800] I [quota.c:3092:quota_parse_limits] 1-vol-quota: /:161061273600
>[2012-05-11 05:36:42.854422] I [io-cache.c:1558:check_cache_size_ok] 1-vol-io-cache: Max cache size is 2104397824
>[2012-05-11 05:36:42.856760] I [io-cache.c:1558:check_cache_size_ok] 1-vol-quick-read: Max cache size is 2104397824
>[2012-05-11 05:36:42.863939] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-0: changing port to 24009 (from 0)
>[2012-05-11 05:36:42.865932] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-3: changing port to 24010 (from 0)
>[2012-05-11 05:36:46.336653] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-1: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:36:46.338618] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-1: Connected to 172.17.251.59:24009, attached to remote volume '/export/brick1'.
>[2012-05-11 05:36:46.338855] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:36:46.339624] I [afr-common.c:3629:afr_notify] 1-vol-replicate-0: Subvolume 'vol-client-1' came back up; going online.
>[2012-05-11 05:36:46.355163] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-1: Server lk version = 1
>[2012-05-11 05:36:46.452497] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-2: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:36:46.454322] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-2: Connected to 172.17.251.60:24009, attached to remote volume '/export/brick2'.
>[2012-05-11 05:36:46.454564] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:36:46.455508] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-2: Server lk version = 1
>[2012-05-11 05:36:46.620304] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-4: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:36:46.627051] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-4: Connected to 172.17.251.59:24010, attached to remote volume '/export/brick4'.
>[2012-05-11 05:36:46.627261] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:36:46.627929] I [afr-common.c:3629:afr_notify] 1-vol-replicate-1: Subvolume 'vol-client-4' came back up; going online.
>[2012-05-11 05:36:46.651668] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-4: Server lk version = 1
>[2012-05-11 05:36:46.735320] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-5: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:36:46.737081] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-5: Connected to 172.17.251.60:24010, attached to remote volume '/export/brick5'.
>[2012-05-11 05:36:46.737291] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:36:46.738176] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-5: Server lk version = 1
>[2012-05-11 05:36:46.913675] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:36:46.917702] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'.
>[2012-05-11 05:36:46.917955] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:36:46.920136] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-0: Server lk version = 1
>[2012-05-11 05:36:47.097094] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:36:47.100719] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'.
>[2012-05-11 05:36:47.100939] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:36:47.211667] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 1
>[2012-05-11 05:36:47.213305] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-3: Server lk version = 1
>[2012-05-11 05:38:56.988094] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-05-11 05:38:59.133193] I [io-cache.c:1558:check_cache_size_ok] 2-vol-quick-read: Max cache size is 2104397824
>[2012-05-11 05:38:59.133836] I [io-cache.c:1558:check_cache_size_ok] 2-vol-io-cache: Max cache size is 2104397824
>[2012-05-11 05:38:59.137816] I [quota.c:3060:quota_parse_limits] 2-vol-quota: /:161061273600
>[2012-05-11 05:38:59.138026] I [quota.c:3092:quota_parse_limits] 2-vol-quota: /:161061273600
>[2012-05-11 05:40:27.615613] I [client.c:2151:notify] 2-vol-client-0: parent translators are ready, attempting connect on transport
>[2012-05-11 05:40:27.714936] I [client.c:2151:notify] 2-vol-client-1: parent translators are ready, attempting connect on transport
>[2012-05-11 05:40:27.815179] I [client.c:2151:notify] 2-vol-client-2: parent translators are ready, attempting connect on transport
>[2012-05-11 05:40:27.982869] I [client.c:2151:notify] 2-vol-client-3: parent translators are ready, attempting connect on transport
>[2012-05-11 05:40:28.184909] I [client.c:2151:notify] 2-vol-client-4: parent translators are ready, attempting connect on transport
>[2012-05-11 05:40:28.380038] I [client.c:2151:notify] 2-vol-client-5: parent translators are ready, attempting connect on transport
>[2012-05-11 05:40:28.580583] I [client.c:2151:notify] 2-vol-client-6: parent translators are ready, attempting connect on transport
>[2012-05-11 05:40:28.781464] I [client.c:2151:notify] 2-vol-client-7: parent translators are ready, attempting connect on transport
>[2012-05-11 05:40:28.981042] I [client.c:2151:notify] 2-vol-client-8: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume vol-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.58
>  4:     option remote-subvolume /export/brick0
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume vol-client-1
>  9:     type protocol/client
> 10:     option remote-host 172.17.251.59
> 11:     option remote-subvolume /export/brick1
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume vol-client-2
> 16:     type protocol/client
> 17:     option remote-host 172.17.251.60
> 18:     option remote-subvolume /export/brick2
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume vol-client-3
> 23:     type protocol/client
> 24:     option remote-host 172.17.251.58
> 25:     option remote-subvolume /export/brick3
> 26:     option transport-type tcp
> 27: end-volume
> 28: 
> 29: volume vol-client-4
> 30:     type protocol/client
> 31:     option remote-host 172.17.251.59
> 32:     option remote-subvolume /export/brick4
> 33:     option transport-type tcp
> 34: end-volume
> 35: 
> 36: volume vol-client-5
> 37:     type protocol/client
> 38:     option remote-host 172.17.251.60
> 39:     option remote-subvolume /export/brick5
> 40:     option transport-type tcp
> 41: end-volume
> 42: 
> 43: volume vol-client-6
> 44:     type protocol/client
> 45:     option remote-host 172.17.251.58
> 46:     option remote-subvolume /export/brick6
> 47:     option transport-type tcp
> 48: end-volume
> 49: 
> 50: volume vol-client-7
> 51:     type protocol/client
> 52:     option remote-host 172.17.251.59
> 53:     option remote-subvolume /export/brick7
> 54:     option transport-type tcp
> 55: end-volume
> 56: 
> 57: volume vol-client-8
> 58:     type protocol/client
> 59:     option remote-host 172.17.251.59
> 60:     option remote-subvolume /export/brick8
> 61:     option transport-type tcp
> 62: end-volume
> 63: 
> 64: volume vol-replicate-0
> 65:     type cluster/replicate
> 66:     subvolumes vol-client-0 vol-client-1 vol-client-2
> 67: end-volume
> 68: 
> 69: volume vol-replicate-1
> 70:     type cluster/replicate
> 71:     subvolumes vol-client-3 vol-client-4 vol-client-5
> 72: end-volume
> 73: 
> 74: volume vol-replicate-2
> 75:     type cluster/replicate
> 76:     subvolumes vol-client-6 vol-client-7 vol-client-8
> 77: end-volume
> 78: 
> 79: volume vol-dht
> 80:     type cluster/distribute
> 81:     subvolumes vol-replicate-0 vol-replicate-1 vol-replicate-2
> 82: end-volume
> 83: 
> 84: volume vol-quota
> 85:     type features/quota
> 86:     option limit-set /:150GB
> 87:     option timeout 0
> 88:     subvolumes vol-dht
> 89: end-volume
> 90: 
> 91: volume vol-write-behind
> 92:     type performance/write-behind
> 93:     subvolumes vol-quota
> 94: end-volume
> 95: 
> 96: volume vol-read-ahead
> 97:     type performance/read-ahead
> 98:     subvolumes vol-write-behind
> 99: end-volume
>100: 
>101: volume vol-io-cache
>102:     type performance/io-cache
>103:     subvolumes vol-read-ahead
>104: end-volume
>105: 
>106: volume vol-quick-read
>107:     type performance/quick-read
>108:     subvolumes vol-io-cache
>109: end-volume
>110: 
>111: volume vol-md-cache
>112:     type performance/md-cache
>113:     subvolumes vol-quick-read
>114: end-volume
>115: 
>116: volume vol
>117:     type debug/io-stats
>118:     option latency-measurement off
>119:     option count-fop-hits off
>120:     subvolumes vol-md-cache
>121: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-05-11 05:40:29.203205] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-0: changing port to 24009 (from 0)
>[2012-05-11 05:40:29.203822] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-1: changing port to 24009 (from 0)
>[2012-05-11 05:40:29.204330] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-3: changing port to 24010 (from 0)
>[2012-05-11 05:40:29.212082] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-7: changing port to 24011 (from 0)
>[2012-05-11 05:40:29.212639] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-6: changing port to 24011 (from 0)
>[2012-05-11 05:40:29.213107] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-4: changing port to 24010 (from 0)
>[2012-05-11 05:40:29.213621] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-2: changing port to 24009 (from 0)
>[2012-05-11 05:40:29.214487] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-8: changing port to 24012 (from 0)
>[2012-05-11 05:40:29.214939] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-5: changing port to 24010 (from 0)
>[2012-05-11 05:40:33.162000] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:33.171726] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'.
>[2012-05-11 05:40:33.172211] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:33.173633] I [afr-common.c:3629:afr_notify] 2-vol-replicate-0: Subvolume 'vol-client-0' came back up; going online.
>[2012-05-11 05:40:33.187651] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-0: Server lk version = 1
>[2012-05-11 05:40:33.645702] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-1: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:33.656032] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-1: Connected to 172.17.251.59:24009, attached to remote volume '/export/brick1'.
>[2012-05-11 05:40:33.656259] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:33.657437] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-1: Server lk version = 1
>[2012-05-11 05:40:33.999599] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:34.001392] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'.
>[2012-05-11 05:40:34.001620] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:34.002301] I [afr-common.c:3629:afr_notify] 2-vol-replicate-1: Subvolume 'vol-client-3' came back up; going online.
>[2012-05-11 05:40:34.002842] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-3: Server lk version = 1
>[2012-05-11 05:40:34.039362] W [socket.c:1521:__socket_proto_state_machine] 0-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009)
>[2012-05-11 05:40:34.039742] I [client.c:2099:client_rpc_notify] 0-vol-client-0: disconnected
>[2012-05-11 05:40:34.048753] W [socket.c:1521:__socket_proto_state_machine] 1-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009)
>[2012-05-11 05:40:34.049082] I [client.c:2099:client_rpc_notify] 1-vol-client-0: disconnected
>[2012-05-11 05:40:34.049434] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009)
>[2012-05-11 05:40:34.049824] I [client.c:2099:client_rpc_notify] 2-vol-client-0: disconnected
>[2012-05-11 05:40:34.384127] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-7: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:34.395302] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-7: Connected to 172.17.251.59:24011, attached to remote volume '/export/brick7'.
>[2012-05-11 05:40:34.395558] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:34.396222] I [afr-common.c:3629:afr_notify] 2-vol-replicate-2: Subvolume 'vol-client-7' came back up; going online.
>[2012-05-11 05:40:34.396744] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-7: Server lk version = 1
>[2012-05-11 05:40:34.730378] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-6: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:34.732434] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-6: Connected to 172.17.251.58:24011, attached to remote volume '/export/brick6'.
>[2012-05-11 05:40:34.732655] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:34.740633] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-6: Server lk version = 1
>[2012-05-11 05:40:35.346168] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-4: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:35.348116] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-4: Connected to 172.17.251.59:24010, attached to remote volume '/export/brick4'.
>[2012-05-11 05:40:35.348401] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:35.361146] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-4: Server lk version = 1
>[2012-05-11 05:40:35.680868] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-2: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:35.683000] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-2: Connected to 172.17.251.60:24009, attached to remote volume '/export/brick2'.
>[2012-05-11 05:40:35.683302] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:35.695776] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-2: Server lk version = 1
>[2012-05-11 05:40:36.015691] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-8: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:36.020227] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-8: Connected to 172.17.251.59:24012, attached to remote volume '/export/brick8'.
>[2012-05-11 05:40:36.020466] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:36.022503] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-8: Server lk version = 1
>[2012-05-11 05:40:36.330214] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-5: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:40:36.332765] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-5: Connected to 172.17.251.60:24010, attached to remote volume '/export/brick5'.
>[2012-05-11 05:40:36.332994] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:40:36.704658] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 2
>[2012-05-11 05:40:36.709974] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-5: Server lk version = 1
>[2012-05-11 05:40:37.384844] W [socket.c:1521:__socket_proto_state_machine] 0-vol-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24010)
>[2012-05-11 05:40:37.393448] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.794282 (xid=0x92902x)
>[2012-05-11 05:40:37.412549] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.422720] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.795266 (xid=0x92903x)
>[2012-05-11 05:40:37.439785] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.443432] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.796214 (xid=0x92904x)
>[2012-05-11 05:40:37.454146] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.461193] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.797145 (xid=0x92905x)
>[2012-05-11 05:40:37.469234] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.471228] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.811409 (xid=0x92906x)
>[2012-05-11 05:40:37.471898] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.482026] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.812499 (xid=0x92907x)
>[2012-05-11 05:40:37.482764] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.484315] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.813305 (xid=0x92908x)
>[2012-05-11 05:40:37.492129] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.493855] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.813941 (xid=0x92909x)
>[2012-05-11 05:40:37.494545] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.500277] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.814570 (xid=0x92910x)
>[2012-05-11 05:40:37.506618] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:40:37.507778] I [client.c:2099:client_rpc_notify] 0-vol-client-3: disconnected
>[2012-05-11 05:40:37.513508] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24010)
>[2012-05-11 05:40:37.514237] I [client.c:2099:client_rpc_notify] 2-vol-client-3: disconnected
>[2012-05-11 05:40:37.514847] W [socket.c:1521:__socket_proto_state_machine] 1-vol-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24010)
>[2012-05-11 05:40:37.520475] I [client.c:2099:client_rpc_notify] 1-vol-client-3: disconnected
>[2012-05-11 05:40:37.963768] E [socket.c:1724:socket_connect_finish] 2-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused)
>[2012-05-11 05:40:41.305768] E [socket.c:1724:socket_connect_finish] 2-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused)
>[2012-05-11 05:40:45.365632] E [socket.c:1724:socket_connect_finish] 0-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused)
>[2012-05-11 05:40:45.715449] E [socket.c:1724:socket_connect_finish] 1-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused)
>[2012-05-11 05:40:48.458590] E [socket.c:1724:socket_connect_finish] 0-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused)
>[2012-05-11 05:40:48.658551] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-6: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24011)
>[2012-05-11 05:40:48.658979] I [client.c:2099:client_rpc_notify] 2-vol-client-6: disconnected
>[2012-05-11 05:40:48.735946] E [socket.c:1724:socket_connect_finish] 1-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused)
>[2012-05-11 05:40:59.725666] E [socket.c:1724:socket_connect_finish] 2-vol-client-6: connection to 172.17.251.58:24011 failed (Connection refused)
>[2012-05-11 05:41:50.440620] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-vol-replicate-1: diff self-heal on /fuse1.1/dir.1/file.3: completed. (0 blocks of 5901 were different (0.00%))
>[2012-05-11 05:41:50.556018] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:41:50.556971] E [afr-self-heal-data.c:1000:afr_sh_common_fxattrop_resp_handler] 0-vol-replicate-1: fxattrop of /fuse1.1/dir.1/file.3 failed on vol-client-3, reason File descriptor in bad state
>[2012-05-11 05:41:50.565323] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-vol-replicate-1: background  data missing-entry gfid self-heal completed on /fuse1.1/dir.1/file.3
>[2012-05-11 05:41:50.592745] W [client3_1-fops.c:3135:client3_1_ftruncate] 0-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:41:50.595233] E [afr-self-heal-data.c:604:afr_sh_data_trim_cbk] 0-vol-replicate-1: ftruncate of /fuse1.1/dir.1/file.3 on subvolume vol-client-3 failed (File descriptor in bad state)
>[2012-05-11 05:41:50.598043] W [client3_1-fops.c:1493:client3_1_inodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:41:50.606027] W [client3_1-fops.c:4049:client3_1_flush] 0-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:41:50.608173] I [afr-self-heal-data.c:115:afr_sh_data_flush_cbk] 0-vol-replicate-1: flush failed on /fuse1.1/dir.1/file.3 on subvolume vol-client-3: File descriptor in bad state
>[2012-05-11 05:41:50.629880] E [afr-self-heal-common.c:2051:afr_self_heal_completion_cbk] 0-vol-replicate-1: background  data missing-entry gfid self-heal failed on /fuse1.1/dir.1/file.3
>[2012-05-11 05:41:50.656965] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 2-vol-replicate-1: added root inode
>[2012-05-11 05:41:50.675040] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 2-vol-replicate-0: added root inode
>[2012-05-11 05:41:50.678886] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 2-vol-replicate-2: added root inode
>[2012-05-11 05:41:51.226959] I [afr-common.c:1347:afr_launch_self_heal] 2-vol-replicate-1: background  meta-data data self-heal triggered. path: , reason: lookup detected pending operations
>[2012-05-11 05:41:51.790044] I [afr-self-heal-data.c:781:afr_sh_data_fix] 2-vol-replicate-1: no active sinks for performing self-heal on file 
>[2012-05-11 05:41:51.827289] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 2-vol-replicate-1: background  meta-data data self-heal completed on 
>[2012-05-11 05:41:51.923854] I [client.c:2160:notify] 0-vol-client-0: current graph is no longer active, destroying rpc_client 
>[2012-05-11 05:41:51.928028] I [client.c:2160:notify] 0-vol-client-1: current graph is no longer active, destroying rpc_client 
>[2012-05-11 05:41:51.930333] I [client.c:2160:notify] 0-vol-client-2: current graph is no longer active, destroying rpc_client 
>[2012-05-11 05:41:51.930709] I [client.c:2160:notify] 0-vol-client-3: current graph is no longer active, destroying rpc_client 
>[2012-05-11 05:41:51.930943] I [client.c:2160:notify] 0-vol-client-4: current graph is no longer active, destroying rpc_client 
>[2012-05-11 05:41:51.931607] I [client.c:2160:notify] 0-vol-client-5: current graph is no longer active, destroying rpc_client 
>[2012-05-11 05:41:51.932522] I [client.c:2099:client_rpc_notify] 0-vol-client-1: disconnected
>[2012-05-11 05:41:51.932971] I [client.c:2099:client_rpc_notify] 0-vol-client-2: disconnected
>[2012-05-11 05:41:51.933466] E [afr-common.c:3666:afr_notify] 0-vol-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-05-11 05:41:51.934680] I [client.c:2099:client_rpc_notify] 0-vol-client-4: disconnected
>[2012-05-11 05:41:51.935027] I [client.c:2099:client_rpc_notify] 0-vol-client-5: disconnected
>[2012-05-11 05:41:51.935238] E [afr-common.c:3666:afr_notify] 0-vol-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-05-11 05:42:26.774298] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:42:26.789596] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'.
>[2012-05-11 05:42:26.789937] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:42:26.792135] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-0: Server lk version = 1
>[2012-05-11 05:42:27.986857] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:42:27.990788] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'.
>[2012-05-11 05:42:27.991069] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:42:27.998980] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-3: Server lk version = 1
>[2012-05-11 05:42:28.103124] I [afr-common.c:1347:afr_launch_self_heal] 2-vol-replicate-1: background  data missing-entry gfid self-heal triggered. path: <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94>, reason: subvolume came online
>[2012-05-11 05:42:28.123597] I [afr-inode-write.c:437:afr_open_fd_fix] 2-vol-replicate-1: Opening fd 0xc09c6dc
>[2012-05-11 05:42:28.124704] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:42:28.125241] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:42:28.125819] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:42:28.126326] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:42:28.131901] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:42:28.132455] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:42:28.132967] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:42:28.133480] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc
>[2012-05-11 05:42:28.144418] I [afr-open.c:327:afr_openfd_fix_open_cbk] 2-vol-replicate-1: fd for <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94> opened successfully on subvolume vol-client-3
>[2012-05-11 05:42:28.176184] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-6: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:42:28.183778] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-6: Connected to 172.17.251.58:24011, attached to remote volume '/export/brick6'.
>[2012-05-11 05:42:28.184072] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:42:28.186269] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-6: Server lk version = 1
>[2012-05-11 05:42:29.493204] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:42:29.503086] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'.
>[2012-05-11 05:42:29.503422] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:42:29.506106] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-3: Server lk version = 1
>[2012-05-11 05:42:29.670397] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330)
>[2012-05-11 05:42:29.674741] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'.
>[2012-05-11 05:42:29.674976] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-11 05:42:29.677481] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-0: Server lk version = 1
>[2012-05-11 05:43:35.972544] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009)
>[2012-05-11 05:43:35.972974] I [client.c:2099:client_rpc_notify] 2-vol-client-0: disconnected
>[2012-05-11 05:43:35.973333] W [socket.c:1521:__socket_proto_state_machine] 1-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009)
>[2012-05-11 05:43:35.973672] I [client.c:2099:client_rpc_notify] 1-vol-client-0: disconnected
>[2012-05-11 05:43:41.314055] W [socket.c:204:__socket_rwv] 2-vol-client-3: writev failed (Connection reset by peer)
>[2012-05-11 05:43:41.327078] I [socket.c:1807:socket_event_handler] 2-transport: disconnecting now
>[2012-05-11 05:43:41.328173] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(INODELK(29)) called at 2012-05-11 05:43:41.294053 (xid=0x40001x)
>[2012-05-11 05:43:41.328565] W [client3_1-fops.c:1493:client3_1_inodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.329265] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.316731 (xid=0x40011x)
>[2012-05-11 05:43:41.329616] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.330170] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.317886 (xid=0x40012x)
>[2012-05-11 05:43:41.330519] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.331062] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.319010 (xid=0x40013x)
>[2012-05-11 05:43:41.331403] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.331954] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.320109 (xid=0x40014x)
>[2012-05-11 05:43:41.332277] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.332863] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.321235 (xid=0x40015x)
>[2012-05-11 05:43:41.333188] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.333757] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.322385 (xid=0x40016x)
>[2012-05-11 05:43:41.334092] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.334649] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.323505 (xid=0x40017x)
>[2012-05-11 05:43:41.334985] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.335546] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.324607 (xid=0x40018x)
>[2012-05-11 05:43:41.335877] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.336441] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.325749 (xid=0x40019x)
>[2012-05-11 05:43:41.336780] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.337317] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(RCHECKSUM(37)) called at 2012-05-11 05:43:41.294963 (xid=0x40002x)
>[2012-05-11 05:43:41.338321] W [client3_1-fops.c:2099:client3_1_rchecksum_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.373978] E [afr-self-heal-algorithm.c:618:sh_diff_checksum_cbk] 2-vol-replicate-1: checksum on <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94> failed on subvolume vol-client-3 (Transport endpoint is not connected)
>[2012-05-11 05:43:41.471816] I [socket.c:2324:socket_submit_request] 2-vol-client-3: not connected (priv->connected = 0)
>[2012-05-11 05:43:41.472855] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40021x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.473526] W [client3_1-fops.c:1493:client3_1_inodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.475926] I [afr-self-heal-algorithm.c:118:sh_loop_driver_done] 2-vol-replicate-1: self-heal aborting on <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94>
>[2012-05-11 05:43:41.476795] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40022x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.477660] W [client3_1-fops.c:882:client3_1_flush_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.477908] I [afr-self-heal-data.c:115:afr_sh_data_flush_cbk] 2-vol-replicate-1: flush failed on <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94> on subvolume vol-client-3: Transport endpoint is not connected
>[2012-05-11 05:43:41.479437] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.297481 (xid=0x40003x)
>[2012-05-11 05:43:41.479805] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.480207] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40023x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.480487] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.481682] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.299476 (xid=0x40004x)
>[2012-05-11 05:43:41.482006] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.482385] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40024x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.482635] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.483742] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.302331 (xid=0x40005x)
>[2012-05-11 05:43:41.484112] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.484492] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40025x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.484751] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.485897] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.304391 (xid=0x40006x)
>[2012-05-11 05:43:41.486212] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.486594] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40026x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.486850] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.488000] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.306735 (xid=0x40007x)
>[2012-05-11 05:43:41.488314] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.488726] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40027x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.488976] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.490088] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.308905 (xid=0x40008x)
>[2012-05-11 05:43:41.490443] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.490811] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40028x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.491062] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.492192] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.310851 (xid=0x40009x)
>[2012-05-11 05:43:41.492539] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.492917] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40029x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.493171] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.494308] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.314395 (xid=0x40010x)
>[2012-05-11 05:43:41.494667] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.495024] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40030x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.495277] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.496421] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.326780 (xid=0x40020x)
>[2012-05-11 05:43:41.496750] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.497107] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40031x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3)
>[2012-05-11 05:43:41.497377] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected
>[2012-05-11 05:43:41.498188] I [client.c:2099:client_rpc_notify] 2-vol-client-3: disconnected
>[2012-05-11 05:43:41.498526] W [socket.c:1521:__socket_proto_state_machine] 1-vol-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24010)
>[2012-05-11 05:43:41.498864] I [client.c:2099:client_rpc_notify] 1-vol-client-3: disconnected
>[2012-05-11 05:43:41.500121] E [socket.c:1724:socket_connect_finish] 2-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused)
>[2012-05-11 05:43:41.508464] E [afr-self-heal-common.c:2051:afr_self_heal_completion_cbk] 2-vol-replicate-1: background  data self-heal failed on <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94>
>[2012-05-11 05:43:41.514761] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:41.515825] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:41.516760] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:41.517693] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:41.518605] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:41.519536] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:41.520478] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:41.521409] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:41.522325] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3:  (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD
>[2012-05-11 05:43:44.426634] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-6: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24011)
>[2012-05-11 05:43:44.427060] I [client.c:2099:client_rpc_notify] 2-vol-client-6: disconnected
>[2012-05-11 05:43:46.890773] E [socket.c:1724:socket_connect_finish] 2-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused)
>[2012-05-11 05:43:47.101102] E [socket.c:1724:socket_connect_finish] 1-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused)
>[2012-05-11 05:43:52.888026] E [socket.c:1724:socket_connect_finish] 1-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused)
>[2012-05-11 05:43:55.600167] E [socket.c:1724:socket_connect_finish] 2-vol-client-6: connection to 172.17.251.58:24011 failed (Connection refused)
>pending frames:
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>frame : type(1) op(WRITE)
>
>patchset: git://git.gluster.com/glusterfs.git
>signal received: 11
>time of crash: 2012-05-11 05:44:08
>configuration details:
>argp 1
>backtrace 1
>dlfcn 1
>fdatasync 1
>libpthread 1
>llistxattr 1
>setfsid 1
>spinlock 1
>epoll.h 1
>xattr.h 1
>st_atim.tv_nsec 1
>package-string: glusterfs 3.3.0qa40
>/lib64/libc.so.6[0x3a89632900]
>/usr/local/lib/libglusterfs.so.0(fd_ctx_dump+0x151)[0x4c57258]
>/usr/local/lib/libglusterfs.so.0(inode_dump+0x1f8)[0x4c41e6a]
>/usr/local/lib/libglusterfs.so.0(inode_table_dump+0x2ac)[0x4c42237]
>/usr/local/lib/libglusterfs.so.0(gf_proc_dump_xlator_info+0x100)[0x4c61457]
>/usr/local/lib/libglusterfs.so.0(gf_proc_dump_info+0x239)[0x4c61ec0]
>/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0x11a)[0x407973]
>/lib64/libpthread.so.0[0x3a89a077f1]
>/lib64/libc.so.6(clone+0x6d)[0x3a896e5ccd]
>---------

Actions: View

Attachments on bug 820887: 583811 | 583812 | 583813