Attachment 696591 Details for Bug 905010 – Glusterfs log

Glusterfs log

glusterfs.log (text/x-log), 49.46 KB, created by Csaba Henk on 2013-02-12 18:36:06 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Csaba Henk

Created: 2013-02-12 18:36:06 UTC

Size: 49.46 KB

patch

obsolete

>[2013-02-12 18:08:45.176449] I [glusterfsd.c:1759:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0.5rhs
>[2013-02-12 18:08:45.176667] D [glusterfsd.c:413:set_fuse_mount_options] 0-: fuse direct io type 2
>[2013-02-12 18:08:45.177010] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins
>[2013-02-12 18:08:45.177055] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so
>[2013-02-12 18:08:45.177186] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-glusterfs: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-02-12 18:08:45.181279] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.36 (port-24007) for hostname: gqac013.sbu.lab.eng.bos.redhat.com and port: 24007
>[2013-02-12 18:08:45.183360] D [io-stats.c:2533:init] 0-master: dangling volume. check volfile 
>[2013-02-12 18:08:45.183378] D [options.c:1045:xlator_option_init_bool] 0-master: option count-fop-hits using set value off
>[2013-02-12 18:08:45.183387] D [options.c:1045:xlator_option_init_bool] 0-master: option latency-measurement using set value off
>[2013-02-12 18:08:45.183463] I [io-cache.c:1549:check_cache_size_ok] 0-master-quick-read: Max cache size is 50734194688
>[2013-02-12 18:08:45.183513] I [io-cache.c:1549:check_cache_size_ok] 0-master-io-cache: Max cache size is 50734194688
>[2013-02-12 18:08:45.185686] D [client.c:2326:client_init_grace_timer] 0-master-client-3: lk-heal = off
>[2013-02-12 18:08:45.185717] D [client.c:2337:client_init_grace_timer] 0-master-client-3: Client grace timeout value = 10
>[2013-02-12 18:08:45.185730] D [options.c:1047:xlator_option_init_path] 0-master-client-3: option remote-subvolume using set value /exportdir/brick4
>[2013-02-12 18:08:45.186220] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-master-client-3: defaulting frame-timeout to 30mins
>[2013-02-12 18:08:45.186239] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so
>[2013-02-12 18:08:45.186273] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-master-client-3: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-02-12 18:08:45.186281] D [client.c:2301:client_init_rpc] 0-master-client-3: client init successful
>[2013-02-12 18:08:45.186289] D [client.c:2326:client_init_grace_timer] 0-master-client-2: lk-heal = off
>[2013-02-12 18:08:45.186295] D [client.c:2337:client_init_grace_timer] 0-master-client-2: Client grace timeout value = 10
>[2013-02-12 18:08:45.186304] D [options.c:1047:xlator_option_init_path] 0-master-client-2: option remote-subvolume using set value /exportdir/brick3
>[2013-02-12 18:08:45.186790] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-master-client-2: defaulting frame-timeout to 30mins
>[2013-02-12 18:08:45.186809] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so
>[2013-02-12 18:08:45.186827] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-master-client-2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-02-12 18:08:45.186835] D [client.c:2301:client_init_rpc] 0-master-client-2: client init successful
>[2013-02-12 18:08:45.186842] D [client.c:2326:client_init_grace_timer] 0-master-client-1: lk-heal = off
>[2013-02-12 18:08:45.186848] D [client.c:2337:client_init_grace_timer] 0-master-client-1: Client grace timeout value = 10
>[2013-02-12 18:08:45.186858] D [options.c:1047:xlator_option_init_path] 0-master-client-1: option remote-subvolume using set value /exportdir/brick2
>[2013-02-12 18:08:45.187339] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-master-client-1: defaulting frame-timeout to 30mins
>[2013-02-12 18:08:45.187353] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so
>[2013-02-12 18:08:45.187369] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-master-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-02-12 18:08:45.187383] D [client.c:2301:client_init_rpc] 0-master-client-1: client init successful
>[2013-02-12 18:08:45.187391] D [client.c:2326:client_init_grace_timer] 0-master-client-0: lk-heal = off
>[2013-02-12 18:08:45.187397] D [client.c:2337:client_init_grace_timer] 0-master-client-0: Client grace timeout value = 10
>[2013-02-12 18:08:45.187406] D [options.c:1047:xlator_option_init_path] 0-master-client-0: option remote-subvolume using set value /exportdir/brick1
>[2013-02-12 18:08:45.187902] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-master-client-0: defaulting frame-timeout to 30mins
>[2013-02-12 18:08:45.187918] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so
>[2013-02-12 18:08:45.187935] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-master-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-02-12 18:08:45.187945] D [client.c:2301:client_init_rpc] 0-master-client-0: client init successful
>[2013-02-12 18:08:45.187960] D [fuse-bridge.c:4239:notify] 0-fuse: got event 12 on graph 0
>[2013-02-12 18:08:45.187979] I [client.c:2150:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2013-02-12 18:08:45.187988] D [name.c:149:client_fill_address_family] 0-master-client-0: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:08:45.190252] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.36 (port-24007) for hostname: 10.16.157.36 and port: 24007
>[2013-02-12 18:08:45.190325] I [client.c:2150:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2013-02-12 18:08:45.190342] D [name.c:149:client_fill_address_family] 0-master-client-1: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:08:45.192567] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.39 (port-24007) for hostname: 10.16.157.39 and port: 24007
>[2013-02-12 18:08:45.192614] I [client.c:2150:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2013-02-12 18:08:45.192626] D [name.c:149:client_fill_address_family] 0-master-client-2: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:08:45.194814] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.42 (port-24007) for hostname: 10.16.157.42 and port: 24007
>[2013-02-12 18:08:45.194849] I [client.c:2150:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>[2013-02-12 18:08:45.194860] D [name.c:149:client_fill_address_family] 0-master-client-3: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:08:45.197041] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.45 (port-24007) for hostname: 10.16.157.45 and port: 24007
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.16.157.36
>  4:     option remote-subvolume /exportdir/brick1
>  5:     option transport-type tcp
>  6:     option username 2e454e16-260c-452c-9747-09bf03566da2
>  7:     option password 8a11be98-4bda-4600-bc5d-3f6a851884df
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 10.16.157.39
> 13:     option remote-subvolume /exportdir/brick2
> 14:     option transport-type tcp
> 15:     option username 2e454e16-260c-452c-9747-09bf03566da2
> 16:     option password 8a11be98-4bda-4600-bc5d-3f6a851884df
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 10.16.157.42
> 22:     option remote-subvolume /exportdir/brick3
> 23:     option transport-type tcp
> 24:     option username 2e454e16-260c-452c-9747-09bf03566da2
> 25:     option password 8a11be98-4bda-4600-bc5d-3f6a851884df
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 10.16.157.45
> 31:     option remote-subvolume /exportdir/brick4
> 32:     option transport-type tcp
> 33:     option username 2e454e16-260c-452c-9747-09bf03566da2
> 34:     option password 8a11be98-4bda-4600-bc5d-3f6a851884df
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2013-02-12 18:08:45.197142] D [glusterfsd-mgmt.c:2095:glusterfs_mgmt_pmap_signin] 0-fsd-mgmt: portmapper signin arguments not given
>[2013-02-12 18:08:45.197169] D [client.c:2051:client_rpc_notify] 0-master-client-0: got RPC_CLNT_CONNECT
>[2013-02-12 18:08:45.197209] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:45.197242] D [client.c:2051:client_rpc_notify] 0-master-client-1: got RPC_CLNT_CONNECT
>[2013-02-12 18:08:45.197274] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:45.197292] D [client.c:2051:client_rpc_notify] 0-master-client-2: got RPC_CLNT_CONNECT
>[2013-02-12 18:08:45.197311] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:45.197336] D [client-handshake.c:1648:server_has_portmap] 0-master-client-0: detected portmapper on server
>[2013-02-12 18:08:45.197362] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:45.197389] D [client.c:2051:client_rpc_notify] 0-master-client-3: got RPC_CLNT_CONNECT
>[2013-02-12 18:08:45.197422] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:45.197453] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2013-02-12 18:08:45.197484] D [socket.c:184:__socket_rwv] 0-master-client-0: EOF from peer 10.16.157.36:24007
>[2013-02-12 18:08:45.197506] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.36:24007)
>[2013-02-12 18:08:45.197522] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-02-12 18:08:45.197539] D [client.c:2116:client_rpc_notify] 0-master-client-0: disconnected (skipped notify)
>[2013-02-12 18:08:45.197554] D [client-handshake.c:1648:server_has_portmap] 0-master-client-1: detected portmapper on server
>[2013-02-12 18:08:45.197577] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:45.197598] D [client-handshake.c:1648:server_has_portmap] 0-master-client-2: detected portmapper on server
>[2013-02-12 18:08:45.197627] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:45.197647] D [client-handshake.c:1648:server_has_portmap] 0-master-client-3: detected portmapper on server
>[2013-02-12 18:08:45.197667] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:45.197866] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-master-client-1: changing port to 49152 (from 0)
>[2013-02-12 18:08:45.197906] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-master-client-2: changing port to 49152 (from 0)
>[2013-02-12 18:08:45.197924] D [socket.c:184:__socket_rwv] 0-master-client-1: EOF from peer 10.16.157.39:24007
>[2013-02-12 18:08:45.197934] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.39:24007)
>[2013-02-12 18:08:45.197940] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-02-12 18:08:45.197954] D [client.c:2116:client_rpc_notify] 0-master-client-1: disconnected (skipped notify)
>[2013-02-12 18:08:45.197963] D [socket.c:184:__socket_rwv] 0-master-client-2: EOF from peer 10.16.157.42:24007
>[2013-02-12 18:08:45.197969] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.42:24007)
>[2013-02-12 18:08:45.197975] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-02-12 18:08:45.197984] D [client.c:2116:client_rpc_notify] 0-master-client-2: disconnected (skipped notify)
>[2013-02-12 18:08:45.197999] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-master-client-3: changing port to 49152 (from 0)
>[2013-02-12 18:08:45.198013] D [socket.c:184:__socket_rwv] 0-master-client-3: EOF from peer 10.16.157.45:24007
>[2013-02-12 18:08:45.198022] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.45:24007)
>[2013-02-12 18:08:45.198028] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-02-12 18:08:45.198037] D [client.c:2116:client_rpc_notify] 0-master-client-3: disconnected (skipped notify)
>[2013-02-12 18:08:49.182194] D [name.c:149:client_fill_address_family] 0-master-client-0: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:08:49.184541] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.36 (port-24007) for hostname: 10.16.157.36 and port: 24007
>[2013-02-12 18:08:49.184616] D [name.c:149:client_fill_address_family] 0-master-client-1: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:08:49.184695] D [client.c:2051:client_rpc_notify] 0-master-client-0: got RPC_CLNT_CONNECT
>[2013-02-12 18:08:49.184790] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:49.184846] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330)
>[2013-02-12 18:08:49.184892] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:49.185033] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-0: clnt-lk-version = 1, server-lk-version = 0
>[2013-02-12 18:08:49.185057] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-0: Connected to 10.16.157.36:24009, attached to remote volume '/exportdir/brick1'.
>[2013-02-12 18:08:49.185070] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-02-12 18:08:49.185082] D [client-handshake.c:1273:client_post_handshake] 0-master-client-0: No fds to open - notifying all parents child up
>[2013-02-12 18:08:49.185105] D [client-handshake.c:489:client_set_lk_version] 0-master-client-0: Sending SET_LK_VERSION
>[2013-02-12 18:08:49.185152] I [afr-common.c:3544:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2013-02-12 18:08:49.185184] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2013-02-12 18:08:49.186804] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.39 (port-24007) for hostname: 10.16.157.39 and port: 24007
>[2013-02-12 18:08:49.186852] D [name.c:149:client_fill_address_family] 0-master-client-2: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:08:49.187082] D [client.c:2051:client_rpc_notify] 0-master-client-1: got RPC_CLNT_CONNECT
>[2013-02-12 18:08:49.187138] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:49.189091] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.42 (port-24007) for hostname: 10.16.157.42 and port: 24007
>[2013-02-12 18:08:49.189129] D [name.c:149:client_fill_address_family] 0-master-client-3: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:08:49.189355] D [client.c:2051:client_rpc_notify] 0-master-client-2: got RPC_CLNT_CONNECT
>[2013-02-12 18:08:49.189408] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:49.191368] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.45 (port-24007) for hostname: 10.16.157.45 and port: 24007
>[2013-02-12 18:08:49.191636] D [client.c:2051:client_rpc_notify] 0-master-client-3: got RPC_CLNT_CONNECT
>[2013-02-12 18:08:49.191693] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:49.339176] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330)
>[2013-02-12 18:08:49.339240] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:08:49.341770] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-2: clnt-lk-version = 1, server-lk-version = 0
>[2013-02-12 18:08:49.341799] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-2: Connected to 10.16.157.42:49152, attached to remote volume '/exportdir/brick3'.
>[2013-02-12 18:08:49.341812] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-02-12 18:08:49.341824] D [client-handshake.c:1273:client_post_handshake] 0-master-client-2: No fds to open - notifying all parents child up
>[2013-02-12 18:08:49.341839] D [client-handshake.c:489:client_set_lk_version] 0-master-client-2: Sending SET_LK_VERSION
>[2013-02-12 18:08:49.341936] I [afr-common.c:3544:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2013-02-12 18:08:49.342961] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2013-02-12 18:09:31.199142] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
>[2013-02-12 18:09:32.199399] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
>[2013-02-12 18:15:43.774919] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330)
>[2013-02-12 18:15:43.775120] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:15:43.781976] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-3: clnt-lk-version = 1, server-lk-version = 0
>[2013-02-12 18:15:43.782004] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-3: Connected to 10.16.157.45:49152, attached to remote volume '/exportdir/brick4'.
>[2013-02-12 18:15:43.782018] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-02-12 18:15:43.782030] D [client-handshake.c:1273:client_post_handshake] 0-master-client-3: No fds to open - notifying all parents child up
>[2013-02-12 18:15:43.782045] D [client-handshake.c:489:client_set_lk_version] 0-master-client-3: Sending SET_LK_VERSION
>[2013-02-12 18:15:43.783775] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2013-02-12 18:15:43.789401] D [dht-diskusage.c:80:dht_du_info_cbk] 0-master-dht: on subvolume 'master-replicate-1': avail_percent is: 50.00 and avail_space is: 200091475968 and avail_inodes is: 99.00
>[2013-02-12 18:16:26.261989] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)
>[2013-02-12 18:16:26.262041] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
>[2013-02-12 18:26:25.178662] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330)
>[2013-02-12 18:26:25.178766] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:26:25.186151] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-1: clnt-lk-version = 1, server-lk-version = 0
>[2013-02-12 18:26:25.186180] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-1: Connected to 10.16.157.39:49152, attached to remote volume '/exportdir/brick2'.
>[2013-02-12 18:26:25.186195] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-02-12 18:26:25.186208] D [client-handshake.c:1273:client_post_handshake] 0-master-client-1: No fds to open - notifying all parents child up
>[2013-02-12 18:26:25.186223] D [client-handshake.c:489:client_set_lk_version] 0-master-client-1: Sending SET_LK_VERSION
>[2013-02-12 18:26:25.186311] D [fuse-bridge.c:4239:notify] 0-fuse: got event 5 on graph 0
>[2013-02-12 18:26:25.189065] I [fuse-bridge.c:4221:fuse_graph_setup] 0-fuse: switched to graph 0
>[2013-02-12 18:26:25.189142] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2013-02-12 18:26:25.189247] D [fuse-bridge.c:3916:fuse_get_mount_status] 0-fuse: mount status is 0
>[2013-02-12 18:26:25.189363] I [fuse-bridge.c:3404:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2013-02-12 18:26:25.189921] D [dht-diskusage.c:80:dht_du_info_cbk] 0-master-dht: on subvolume 'master-replicate-0': avail_percent is: 98.00 and avail_space is: 387467943936 and avail_inodes is: 99.00
>[2013-02-12 18:27:07.355784] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
>[2013-02-12 18:27:07.355853] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
>[2013-02-12 18:27:07.355871] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-master-client-2: server 10.16.157.42:49152 has not responded in the last 42 seconds, disconnecting.
>[2013-02-12 18:27:07.355947] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-master-client-3: server 10.16.157.45:49152 has not responded in the last 42 seconds, disconnecting.
>[2013-02-12 18:27:07.356016] D [socket.c:184:__socket_rwv] 0-master-client-2: EOF from peer 10.16.157.42:49152
>[2013-02-12 18:27:07.356074] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.42:49152)
>[2013-02-12 18:27:07.356095] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-02-12 18:27:07.356272] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-2: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:26:25.189476 (xid=0x9x)
>[2013-02-12 18:27:07.356296] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2013-02-12 18:27:07.356324] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-2: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-12 18:26:25.189484 (xid=0x10x)
>[2013-02-12 18:27:07.356334] W [client-handshake.c:275:client_ping_cbk] 0-master-client-2: timer must have expired
>[2013-02-12 18:27:07.356343] I [client.c:2098:client_rpc_notify] 0-master-client-2: disconnected
>[2013-02-12 18:27:07.356354] D [fuse-bridge.c:4239:notify] 0-fuse: got event 8 on graph 0
>[2013-02-12 18:27:07.356363] D [socket.c:184:__socket_rwv] 0-master-client-3: EOF from peer 10.16.157.45:49152
>[2013-02-12 18:27:07.356371] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.45:49152)
>[2013-02-12 18:27:07.356377] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-02-12 18:27:07.356403] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:26:25.189498 (xid=0x8x)
>[2013-02-12 18:27:07.356415] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2013-02-12 18:27:07.356534] D [name.c:149:client_fill_address_family] 0-master-client-3: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:27:07.358841] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.45 (port-24007) for hostname: 10.16.157.45 and port: 24007
>[2013-02-12 18:27:07.358886] I [socket.c:2315:socket_submit_request] 0-master-client-3: not connected (priv->connected = 0)
>[2013-02-12 18:27:07.358897] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-master-client-3: failed to submit rpc-request (XID: 0x10x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (master-client-3)
>[2013-02-12 18:27:07.358908] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2013-02-12 18:27:07.358920] D [dht-common.c:434:dht_lookup_dir_cbk] 0-master-dht: lookup of / on master-replicate-1 returned error (Transport endpoint is not connected)
>[2013-02-12 18:27:07.358932] D [client.c:234:client_submit_request] 0-master-client-3: rpc_clnt_submit failed
>[2013-02-12 18:27:07.358964] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-3: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-12 18:26:25.189504 (xid=0x9x)
>[2013-02-12 18:27:07.358978] W [client-handshake.c:275:client_ping_cbk] 0-master-client-3: timer must have expired
>[2013-02-12 18:27:07.358986] I [client.c:2098:client_rpc_notify] 0-master-client-3: disconnected
>[2013-02-12 18:27:07.358992] E [afr-common.c:3581:afr_notify] 0-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2013-02-12 18:27:07.359000] D [fuse-bridge.c:4239:notify] 0-fuse: got event 6 on graph 0
>[2013-02-12 18:27:07.359034] I [afr-common.c:1976:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2013-02-12 18:27:07.359071] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 240 3359 ]
>[2013-02-12 18:27:07.359080] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 6342 7 ]
>[2013-02-12 18:27:07.359088] D [afr-self-heal-common.c:829:afr_mark_sources] 0-master-replicate-0: Number of sources: 1
>[2013-02-12 18:27:07.359095] D [afr-self-heal-data.c:863:afr_lookup_select_read_child_by_txn_type] 0-master-replicate-0: returning read_child: 1
>[2013-02-12 18:27:07.359102] D [afr-common.c:1294:afr_lookup_select_read_child] 0-master-replicate-0: Source selected as 1 for /
>[2013-02-12 18:27:07.359113] D [afr-common.c:1097:afr_lookup_build_response_params] 0-master-replicate-0: Building lookup response from 1
>[2013-02-12 18:27:07.359123] D [afr-common.c:1158:afr_lookup_set_self_heal_params_by_xattr] 0-master-replicate-0: entry self-heal is pending for /.
>[2013-02-12 18:27:07.359130] D [afr-common.c:1158:afr_lookup_set_self_heal_params_by_xattr] 0-master-replicate-0: entry self-heal is pending for /.
>[2013-02-12 18:27:07.359139] D [afr-common.c:1340:afr_launch_self_heal] 0-master-replicate-0: background  entry self-heal triggered. path: /, reason: lookup detected pending operations
>[2013-02-12 18:27:07.359156] D [afr-self-heal-metadata.c:63:afr_sh_metadata_done] 0-master-replicate-0: proceeding to entry check on /
>[2013-02-12 18:27:07.359204] D [client.c:2051:client_rpc_notify] 0-master-client-3: got RPC_CLNT_CONNECT
>[2013-02-12 18:27:07.359226] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:27:07.359919] D [afr-self-heal-entry.c:2384:afr_sh_post_nonblocking_entry_cbk] 0-master-replicate-0: Non Blocking entrylks done for /. Proceeding to FOP
>[2013-02-12 18:27:07.359956] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up / on subvolume master-client-0
>[2013-02-12 18:27:07.360002] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up / on subvolume master-client-1
>[2013-02-12 18:27:07.361084] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 240 3359 ]
>[2013-02-12 18:27:07.361109] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 6342 7 ]
>[2013-02-12 18:27:07.361122] D [afr-self-heal-common.c:829:afr_mark_sources] 0-master-replicate-0: Number of sources: 1
>[2013-02-12 18:27:07.361134] I [afr-self-heal-entry.c:2335:afr_sh_entry_fix] 0-master-replicate-0: /: Performing conservative merge
>[2013-02-12 18:27:07.361149] D [afr-self-heal-entry.c:2291:afr_sh_entry_sync_prepare] 0-master-replicate-0: no active sources for / found. merging all entries as a conservative decision
>[2013-02-12 18:27:07.362417] D [afr-self-heal-entry.c:850:afr_sh_entry_expunge_all] 0-master-replicate-0: no active sources for / to expunge entries
>[2013-02-12 18:27:07.362446] D [afr-self-heal-entry.c:2076:afr_sh_entry_impunge_subvol] 0-master-replicate-0: /: readdir from offset 0
>[2013-02-12 18:27:07.362852] D [afr-self-heal-entry.c:2050:afr_sh_entry_impunge_readdir_cbk] 0-master-replicate-0: readdir'ed 11 entries from master-client-0
>[2013-02-12 18:27:07.362911] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /8ff on subvolume master-client-0
>[2013-02-12 18:27:07.362961] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /8ff on subvolume master-client-1
>[2013-02-12 18:27:07.362995] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /940 on subvolume master-client-0
>[2013-02-12 18:27:07.363017] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /940 on subvolume master-client-1
>[2013-02-12 18:27:07.363044] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /970 on subvolume master-client-0
>[2013-02-12 18:27:07.363065] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /970 on subvolume master-client-1
>[2013-02-12 18:27:07.363096] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /99e on subvolume master-client-0
>[2013-02-12 18:27:07.363123] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /99e on subvolume master-client-1
>[2013-02-12 18:27:07.363153] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /9ca on subvolume master-client-0
>[2013-02-12 18:27:07.363186] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /9ca on subvolume master-client-1
>[2013-02-12 18:27:07.363218] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /a1c on subvolume master-client-0
>[2013-02-12 18:27:07.363242] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /a1c on subvolume master-client-1
>[2013-02-12 18:27:07.363271] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /a68 on subvolume master-client-0
>[2013-02-12 18:27:07.363300] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /a68 on subvolume master-client-1
>[2013-02-12 18:27:07.363339] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /perf-test on subvolume master-client-0
>[2013-02-12 18:27:07.363365] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /perf-test on subvolume master-client-1
>[2013-02-12 18:27:07.363394] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /linux2 on subvolume master-client-0
>[2013-02-12 18:27:07.363419] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /linux2 on subvolume master-client-1
>[2013-02-12 18:27:17.357425] D [name.c:149:client_fill_address_family] 0-master-client-2: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:27:17.359762] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.42 (port-24007) for hostname: 10.16.157.42 and port: 24007
>[2013-02-12 18:27:17.359963] D [client.c:2051:client_rpc_notify] 0-master-client-2: got RPC_CLNT_CONNECT
>[2013-02-12 18:27:17.360048] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:27:49.364569] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
>[2013-02-12 18:28:31.370972] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-master-client-1: server 10.16.157.39:49152 has not responded in the last 42 seconds, disconnecting.
>[2013-02-12 18:28:31.371141] D [socket.c:184:__socket_rwv] 0-master-client-1: EOF from peer 10.16.157.39:49152
>[2013-02-12 18:28:31.371198] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.39:49152)
>[2013-02-12 18:28:31.371219] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-02-12 18:28:31.371289] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.362985 (xid=0x13x)
>[2013-02-12 18:28:31.371324] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /8ff (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371335] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /8ff on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371379] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363035 (xid=0x14x)
>[2013-02-12 18:28:31.371391] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /940 (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371399] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /940 on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371424] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363085 (xid=0x15x)
>[2013-02-12 18:28:31.371435] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /970 (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371442] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /970 on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371467] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363144 (xid=0x16x)
>[2013-02-12 18:28:31.371478] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /99e (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371485] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /99e on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371510] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363206 (xid=0x17x)
>[2013-02-12 18:28:31.371521] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /9ca (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371528] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /9ca on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371553] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363262 (xid=0x18x)
>[2013-02-12 18:28:31.371567] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /a1c (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371575] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /a1c on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371600] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363327 (xid=0x19x)
>[2013-02-12 18:28:31.371611] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /a68 (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371619] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /a68 on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371643] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363385 (xid=0x20x)
>[2013-02-12 18:28:31.371654] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /perf-test (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371661] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /perf-test on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371686] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363440 (xid=0x21x)
>[2013-02-12 18:28:31.371697] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /linux2 (00000000-0000-0000-0000-000000000000)
>[2013-02-12 18:28:31.371704] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /linux2 on subvolume master-client-1 => -1 (Transport endpoint is not connected)
>[2013-02-12 18:28:31.371715] D [afr-self-heal-entry.c:2076:afr_sh_entry_impunge_subvol] 0-master-replicate-0: /: readdir from offset 2147483647
>[2013-02-12 18:28:31.371802] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-12 18:27:49.364662 (xid=0x22x)
>[2013-02-12 18:28:31.371906] W [client-handshake.c:275:client_ping_cbk] 0-master-client-1: timer must have expired
>[2013-02-12 18:28:31.371916] I [client.c:2098:client_rpc_notify] 0-master-client-1: disconnected
>[2013-02-12 18:28:31.371929] D [fuse-bridge.c:4239:notify] 0-fuse: got event 8 on graph 0
>[2013-02-12 18:28:31.371950] D [afr-self-heal-entry.c:2076:afr_sh_entry_impunge_subvol] 0-master-replicate-0: /: readdir from offset 0
>[2013-02-12 18:28:31.371965] W [client3_1-fops.c:5652:client3_1_readdirp] 0-master-client-1:  (00000000-0000-0000-0000-000000000001) remote_fd is -1. EBADFD
>[2013-02-12 18:28:31.371974] I [afr-self-heal-entry.c:2030:afr_sh_entry_impunge_readdir_cbk] 0-master-replicate-0: readdir of / on subvolume master-client-1 failed (File descriptor in bad state)
>[2013-02-12 18:28:31.371981] D [afr-lk-common.c:408:transaction_lk_op] 0-master-replicate-0: lk op is for a self heal
>[2013-02-12 18:28:31.372009] D [client.c:180:client_submit_request] 0-master-client-1: connection in disconnected state
>[2013-02-12 18:28:31.372025] W [client3_1-fops.c:1655:client3_1_entrylk_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected
>[2013-02-12 18:28:31.372076] E [afr-lk-common.c:703:afr_unlock_entrylk_cbk] 0-master-replicate-0: /: unlock failed on 1, reason: Transport endpoint is not connected
>[2013-02-12 18:28:31.372101] E [afr-self-heal-common.c:2162:afr_self_heal_completion_cbk] 0-master-replicate-0: background  entry self-heal failed on /
>[2013-02-12 18:28:31.372118] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 240 3359 ]
>[2013-02-12 18:28:31.372126] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 6342 7 ]
>[2013-02-12 18:28:31.372133] D [afr-self-heal-common.c:829:afr_mark_sources] 0-master-replicate-0: Number of sources: 1
>[2013-02-12 18:28:31.372140] D [afr-self-heal-data.c:863:afr_lookup_select_read_child_by_txn_type] 0-master-replicate-0: returning read_child: 1
>[2013-02-12 18:28:31.372147] D [afr-common.c:1294:afr_lookup_select_read_child] 0-master-replicate-0: Source selected as 1 for /
>[2013-02-12 18:28:31.372156] D [afr-common.c:1097:afr_lookup_build_response_params] 0-master-replicate-0: Building lookup response from 1
>[2013-02-12 18:28:31.372170] I [dht-layout.c:598:dht_layout_normalize] 0-master-dht: found anomalies in /. holes=1 overlaps=0
>[2013-02-12 18:28:31.372218] D [dht-layout.c:614:dht_layout_normalize] (-->/usr/lib64/glusterfs/3.3.0.5rhs/xlator/cluster/replicate.so(afr_self_heal_completion_cbk+0x120) [0x7f962c6cff70] (-->/usr/lib64/glusterfs/3.3.0.5rhs/xlator/cluster/replicate.so(afr_self_heal_lookup_unwind+0x178) [0x7f962c6f1b68] (-->/usr/lib64/glusterfs/3.3.0.5rhs/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x5dc) [0x7f962c47e73c]))) 0-master-dht: path=/ err=Transport endpoint is not connected on subvol=master-replicate-1
>[2013-02-12 18:28:31.372230] D [dht-common.c:482:dht_lookup_dir_cbk] 0-master-dht: fixing assignment on /
>[2013-02-12 18:28:31.372238] W [dht-selfheal.c:872:dht_selfheal_directory] 0-master-dht: 1 subvolumes down -- not fixing
>[2013-02-12 18:28:31.372286] D [client3_1-fops.c:2806:client_fdctx_destroy] 0-master-client-0: sending releasedir on fd
>[2013-02-12 18:28:41.372585] D [name.c:149:client_fill_address_family] 0-master-client-1: address-family not specified, guessing it to be inet/inet6
>[2013-02-12 18:28:41.375044] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.39 (port-24007) for hostname: 10.16.157.39 and port: 24007
>[2013-02-12 18:28:41.375343] D [client.c:2051:client_rpc_notify] 0-master-client-1: got RPC_CLNT_CONNECT
>[2013-02-12 18:28:41.375398] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:29:13.379831] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
>[2013-02-12 18:29:26.729592] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330)
>[2013-02-12 18:29:26.729669] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-02-12 18:29:26.731436] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-1: clnt-lk-version = 1, server-lk-version = 0
>[2013-02-12 18:29:26.731475] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-1: Connected to 10.16.157.39:49152, attached to remote volume '/exportdir/brick2'.
>[2013-02-12 18:29:26.731489] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-02-12 18:29:26.731501] I [client-handshake.c:1260:client_post_handshake] 0-master-client-1: 1 fds open - Delaying child_up until they are re-opened
>[2013-02-12 18:29:26.731524] D [client-handshake.c:1122:protocol_client_reopendir] 0-master-client-1: attempting reopen on <gfid:00000000-0000-0000-0000-000000000001>
>[2013-02-12 18:29:26.733431] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-master-client-1: reopendir on <gfid:00000000-0000-0000-0000-000000000001> succeeded (fd = 0)
>[2013-02-12 18:29:26.733462] I [client-lk.c:601:decrement_reopen_fd_count] 0-master-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2013-02-12 18:29:26.733486] D [client-handshake.c:489:client_set_lk_version] 0-master-client-1: Sending SET_LK_VERSION
>[2013-02-12 18:29:26.733525] D [fuse-bridge.c:4239:notify] 0-fuse: got event 8 on graph 0
>[2013-02-12 18:29:26.733545] D [client3_1-fops.c:2806:client_fdctx_destroy] 0-master-client-1: sending releasedir on fd
>[2013-02-12 18:29:26.734523] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1

Actions: View

Attachments on bug 905010: 688905 | 693286 | 696591