Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 696591 Details for
Bug 905010
Geo-rep session got killed by itself with rsync process in "D" state
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Glusterfs log
glusterfs.log (text/x-log), 49.46 KB, created by
Csaba Henk
on 2013-02-12 18:36:06 UTC
(
hide
)
Description:
Glusterfs log
Filename:
MIME Type:
Creator:
Csaba Henk
Created:
2013-02-12 18:36:06 UTC
Size:
49.46 KB
patch
obsolete
>[2013-02-12 18:08:45.176449] I [glusterfsd.c:1759:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0.5rhs >[2013-02-12 18:08:45.176667] D [glusterfsd.c:413:set_fuse_mount_options] 0-: fuse direct io type 2 >[2013-02-12 18:08:45.177010] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins >[2013-02-12 18:08:45.177055] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so >[2013-02-12 18:08:45.177186] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-glusterfs: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-02-12 18:08:45.181279] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.36 (port-24007) for hostname: gqac013.sbu.lab.eng.bos.redhat.com and port: 24007 >[2013-02-12 18:08:45.183360] D [io-stats.c:2533:init] 0-master: dangling volume. check volfile >[2013-02-12 18:08:45.183378] D [options.c:1045:xlator_option_init_bool] 0-master: option count-fop-hits using set value off >[2013-02-12 18:08:45.183387] D [options.c:1045:xlator_option_init_bool] 0-master: option latency-measurement using set value off >[2013-02-12 18:08:45.183463] I [io-cache.c:1549:check_cache_size_ok] 0-master-quick-read: Max cache size is 50734194688 >[2013-02-12 18:08:45.183513] I [io-cache.c:1549:check_cache_size_ok] 0-master-io-cache: Max cache size is 50734194688 >[2013-02-12 18:08:45.185686] D [client.c:2326:client_init_grace_timer] 0-master-client-3: lk-heal = off >[2013-02-12 18:08:45.185717] D [client.c:2337:client_init_grace_timer] 0-master-client-3: Client grace timeout value = 10 >[2013-02-12 18:08:45.185730] D [options.c:1047:xlator_option_init_path] 0-master-client-3: option remote-subvolume using set value /exportdir/brick4 >[2013-02-12 18:08:45.186220] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-master-client-3: defaulting frame-timeout to 30mins >[2013-02-12 18:08:45.186239] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so >[2013-02-12 18:08:45.186273] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-master-client-3: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-02-12 18:08:45.186281] D [client.c:2301:client_init_rpc] 0-master-client-3: client init successful >[2013-02-12 18:08:45.186289] D [client.c:2326:client_init_grace_timer] 0-master-client-2: lk-heal = off >[2013-02-12 18:08:45.186295] D [client.c:2337:client_init_grace_timer] 0-master-client-2: Client grace timeout value = 10 >[2013-02-12 18:08:45.186304] D [options.c:1047:xlator_option_init_path] 0-master-client-2: option remote-subvolume using set value /exportdir/brick3 >[2013-02-12 18:08:45.186790] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-master-client-2: defaulting frame-timeout to 30mins >[2013-02-12 18:08:45.186809] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so >[2013-02-12 18:08:45.186827] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-master-client-2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-02-12 18:08:45.186835] D [client.c:2301:client_init_rpc] 0-master-client-2: client init successful >[2013-02-12 18:08:45.186842] D [client.c:2326:client_init_grace_timer] 0-master-client-1: lk-heal = off >[2013-02-12 18:08:45.186848] D [client.c:2337:client_init_grace_timer] 0-master-client-1: Client grace timeout value = 10 >[2013-02-12 18:08:45.186858] D [options.c:1047:xlator_option_init_path] 0-master-client-1: option remote-subvolume using set value /exportdir/brick2 >[2013-02-12 18:08:45.187339] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-master-client-1: defaulting frame-timeout to 30mins >[2013-02-12 18:08:45.187353] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so >[2013-02-12 18:08:45.187369] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-master-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-02-12 18:08:45.187383] D [client.c:2301:client_init_rpc] 0-master-client-1: client init successful >[2013-02-12 18:08:45.187391] D [client.c:2326:client_init_grace_timer] 0-master-client-0: lk-heal = off >[2013-02-12 18:08:45.187397] D [client.c:2337:client_init_grace_timer] 0-master-client-0: Client grace timeout value = 10 >[2013-02-12 18:08:45.187406] D [options.c:1047:xlator_option_init_path] 0-master-client-0: option remote-subvolume using set value /exportdir/brick1 >[2013-02-12 18:08:45.187902] D [rpc-clnt.c:973:rpc_clnt_connection_init] 0-master-client-0: defaulting frame-timeout to 30mins >[2013-02-12 18:08:45.187918] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.3.0.5rhs/rpc-transport/socket.so >[2013-02-12 18:08:45.187935] D [rpc-clnt.c:1379:rpcclnt_cbk_program_register] 0-master-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-02-12 18:08:45.187945] D [client.c:2301:client_init_rpc] 0-master-client-0: client init successful >[2013-02-12 18:08:45.187960] D [fuse-bridge.c:4239:notify] 0-fuse: got event 12 on graph 0 >[2013-02-12 18:08:45.187979] I [client.c:2150:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2013-02-12 18:08:45.187988] D [name.c:149:client_fill_address_family] 0-master-client-0: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:08:45.190252] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.36 (port-24007) for hostname: 10.16.157.36 and port: 24007 >[2013-02-12 18:08:45.190325] I [client.c:2150:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2013-02-12 18:08:45.190342] D [name.c:149:client_fill_address_family] 0-master-client-1: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:08:45.192567] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.39 (port-24007) for hostname: 10.16.157.39 and port: 24007 >[2013-02-12 18:08:45.192614] I [client.c:2150:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2013-02-12 18:08:45.192626] D [name.c:149:client_fill_address_family] 0-master-client-2: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:08:45.194814] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.42 (port-24007) for hostname: 10.16.157.42 and port: 24007 >[2013-02-12 18:08:45.194849] I [client.c:2150:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >[2013-02-12 18:08:45.194860] D [name.c:149:client_fill_address_family] 0-master-client-3: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:08:45.197041] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.45 (port-24007) for hostname: 10.16.157.45 and port: 24007 >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 10.16.157.36 > 4: option remote-subvolume /exportdir/brick1 > 5: option transport-type tcp > 6: option username 2e454e16-260c-452c-9747-09bf03566da2 > 7: option password 8a11be98-4bda-4600-bc5d-3f6a851884df > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 10.16.157.39 > 13: option remote-subvolume /exportdir/brick2 > 14: option transport-type tcp > 15: option username 2e454e16-260c-452c-9747-09bf03566da2 > 16: option password 8a11be98-4bda-4600-bc5d-3f6a851884df > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 10.16.157.42 > 22: option remote-subvolume /exportdir/brick3 > 23: option transport-type tcp > 24: option username 2e454e16-260c-452c-9747-09bf03566da2 > 25: option password 8a11be98-4bda-4600-bc5d-3f6a851884df > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 10.16.157.45 > 31: option remote-subvolume /exportdir/brick4 > 32: option transport-type tcp > 33: option username 2e454e16-260c-452c-9747-09bf03566da2 > 34: option password 8a11be98-4bda-4600-bc5d-3f6a851884df > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2013-02-12 18:08:45.197142] D [glusterfsd-mgmt.c:2095:glusterfs_mgmt_pmap_signin] 0-fsd-mgmt: portmapper signin arguments not given >[2013-02-12 18:08:45.197169] D [client.c:2051:client_rpc_notify] 0-master-client-0: got RPC_CLNT_CONNECT >[2013-02-12 18:08:45.197209] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:45.197242] D [client.c:2051:client_rpc_notify] 0-master-client-1: got RPC_CLNT_CONNECT >[2013-02-12 18:08:45.197274] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:45.197292] D [client.c:2051:client_rpc_notify] 0-master-client-2: got RPC_CLNT_CONNECT >[2013-02-12 18:08:45.197311] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:45.197336] D [client-handshake.c:1648:server_has_portmap] 0-master-client-0: detected portmapper on server >[2013-02-12 18:08:45.197362] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:45.197389] D [client.c:2051:client_rpc_notify] 0-master-client-3: got RPC_CLNT_CONNECT >[2013-02-12 18:08:45.197422] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:45.197453] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2013-02-12 18:08:45.197484] D [socket.c:184:__socket_rwv] 0-master-client-0: EOF from peer 10.16.157.36:24007 >[2013-02-12 18:08:45.197506] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.36:24007) >[2013-02-12 18:08:45.197522] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2013-02-12 18:08:45.197539] D [client.c:2116:client_rpc_notify] 0-master-client-0: disconnected (skipped notify) >[2013-02-12 18:08:45.197554] D [client-handshake.c:1648:server_has_portmap] 0-master-client-1: detected portmapper on server >[2013-02-12 18:08:45.197577] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:45.197598] D [client-handshake.c:1648:server_has_portmap] 0-master-client-2: detected portmapper on server >[2013-02-12 18:08:45.197627] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:45.197647] D [client-handshake.c:1648:server_has_portmap] 0-master-client-3: detected portmapper on server >[2013-02-12 18:08:45.197667] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:45.197866] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-master-client-1: changing port to 49152 (from 0) >[2013-02-12 18:08:45.197906] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-master-client-2: changing port to 49152 (from 0) >[2013-02-12 18:08:45.197924] D [socket.c:184:__socket_rwv] 0-master-client-1: EOF from peer 10.16.157.39:24007 >[2013-02-12 18:08:45.197934] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.39:24007) >[2013-02-12 18:08:45.197940] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2013-02-12 18:08:45.197954] D [client.c:2116:client_rpc_notify] 0-master-client-1: disconnected (skipped notify) >[2013-02-12 18:08:45.197963] D [socket.c:184:__socket_rwv] 0-master-client-2: EOF from peer 10.16.157.42:24007 >[2013-02-12 18:08:45.197969] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.42:24007) >[2013-02-12 18:08:45.197975] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2013-02-12 18:08:45.197984] D [client.c:2116:client_rpc_notify] 0-master-client-2: disconnected (skipped notify) >[2013-02-12 18:08:45.197999] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-master-client-3: changing port to 49152 (from 0) >[2013-02-12 18:08:45.198013] D [socket.c:184:__socket_rwv] 0-master-client-3: EOF from peer 10.16.157.45:24007 >[2013-02-12 18:08:45.198022] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.45:24007) >[2013-02-12 18:08:45.198028] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2013-02-12 18:08:45.198037] D [client.c:2116:client_rpc_notify] 0-master-client-3: disconnected (skipped notify) >[2013-02-12 18:08:49.182194] D [name.c:149:client_fill_address_family] 0-master-client-0: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:08:49.184541] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.36 (port-24007) for hostname: 10.16.157.36 and port: 24007 >[2013-02-12 18:08:49.184616] D [name.c:149:client_fill_address_family] 0-master-client-1: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:08:49.184695] D [client.c:2051:client_rpc_notify] 0-master-client-0: got RPC_CLNT_CONNECT >[2013-02-12 18:08:49.184790] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:49.184846] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330) >[2013-02-12 18:08:49.184892] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:49.185033] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-0: clnt-lk-version = 1, server-lk-version = 0 >[2013-02-12 18:08:49.185057] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-0: Connected to 10.16.157.36:24009, attached to remote volume '/exportdir/brick1'. >[2013-02-12 18:08:49.185070] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2013-02-12 18:08:49.185082] D [client-handshake.c:1273:client_post_handshake] 0-master-client-0: No fds to open - notifying all parents child up >[2013-02-12 18:08:49.185105] D [client-handshake.c:489:client_set_lk_version] 0-master-client-0: Sending SET_LK_VERSION >[2013-02-12 18:08:49.185152] I [afr-common.c:3544:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2013-02-12 18:08:49.185184] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2013-02-12 18:08:49.186804] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.39 (port-24007) for hostname: 10.16.157.39 and port: 24007 >[2013-02-12 18:08:49.186852] D [name.c:149:client_fill_address_family] 0-master-client-2: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:08:49.187082] D [client.c:2051:client_rpc_notify] 0-master-client-1: got RPC_CLNT_CONNECT >[2013-02-12 18:08:49.187138] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:49.189091] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.42 (port-24007) for hostname: 10.16.157.42 and port: 24007 >[2013-02-12 18:08:49.189129] D [name.c:149:client_fill_address_family] 0-master-client-3: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:08:49.189355] D [client.c:2051:client_rpc_notify] 0-master-client-2: got RPC_CLNT_CONNECT >[2013-02-12 18:08:49.189408] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:49.191368] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.45 (port-24007) for hostname: 10.16.157.45 and port: 24007 >[2013-02-12 18:08:49.191636] D [client.c:2051:client_rpc_notify] 0-master-client-3: got RPC_CLNT_CONNECT >[2013-02-12 18:08:49.191693] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:49.339176] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330) >[2013-02-12 18:08:49.339240] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:08:49.341770] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-2: clnt-lk-version = 1, server-lk-version = 0 >[2013-02-12 18:08:49.341799] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-2: Connected to 10.16.157.42:49152, attached to remote volume '/exportdir/brick3'. >[2013-02-12 18:08:49.341812] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2013-02-12 18:08:49.341824] D [client-handshake.c:1273:client_post_handshake] 0-master-client-2: No fds to open - notifying all parents child up >[2013-02-12 18:08:49.341839] D [client-handshake.c:489:client_set_lk_version] 0-master-client-2: Sending SET_LK_VERSION >[2013-02-12 18:08:49.341936] I [afr-common.c:3544:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2013-02-12 18:08:49.342961] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2013-02-12 18:09:31.199142] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) >[2013-02-12 18:09:32.199399] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (0 || 0) >[2013-02-12 18:15:43.774919] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330) >[2013-02-12 18:15:43.775120] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:15:43.781976] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-3: clnt-lk-version = 1, server-lk-version = 0 >[2013-02-12 18:15:43.782004] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-3: Connected to 10.16.157.45:49152, attached to remote volume '/exportdir/brick4'. >[2013-02-12 18:15:43.782018] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2013-02-12 18:15:43.782030] D [client-handshake.c:1273:client_post_handshake] 0-master-client-3: No fds to open - notifying all parents child up >[2013-02-12 18:15:43.782045] D [client-handshake.c:489:client_set_lk_version] 0-master-client-3: Sending SET_LK_VERSION >[2013-02-12 18:15:43.783775] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2013-02-12 18:15:43.789401] D [dht-diskusage.c:80:dht_du_info_cbk] 0-master-dht: on subvolume 'master-replicate-1': avail_percent is: 50.00 and avail_space is: 200091475968 and avail_inodes is: 99.00 >[2013-02-12 18:16:26.261989] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (0 || 0) >[2013-02-12 18:16:26.262041] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (0 || 0) >[2013-02-12 18:26:25.178662] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330) >[2013-02-12 18:26:25.178766] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:26:25.186151] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-1: clnt-lk-version = 1, server-lk-version = 0 >[2013-02-12 18:26:25.186180] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-1: Connected to 10.16.157.39:49152, attached to remote volume '/exportdir/brick2'. >[2013-02-12 18:26:25.186195] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2013-02-12 18:26:25.186208] D [client-handshake.c:1273:client_post_handshake] 0-master-client-1: No fds to open - notifying all parents child up >[2013-02-12 18:26:25.186223] D [client-handshake.c:489:client_set_lk_version] 0-master-client-1: Sending SET_LK_VERSION >[2013-02-12 18:26:25.186311] D [fuse-bridge.c:4239:notify] 0-fuse: got event 5 on graph 0 >[2013-02-12 18:26:25.189065] I [fuse-bridge.c:4221:fuse_graph_setup] 0-fuse: switched to graph 0 >[2013-02-12 18:26:25.189142] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2013-02-12 18:26:25.189247] D [fuse-bridge.c:3916:fuse_get_mount_status] 0-fuse: mount status is 0 >[2013-02-12 18:26:25.189363] I [fuse-bridge.c:3404:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 >[2013-02-12 18:26:25.189921] D [dht-diskusage.c:80:dht_du_info_cbk] 0-master-dht: on subvolume 'master-replicate-0': avail_percent is: 98.00 and avail_space is: 387467943936 and avail_inodes is: 99.00 >[2013-02-12 18:27:07.355784] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) >[2013-02-12 18:27:07.355853] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) >[2013-02-12 18:27:07.355871] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-master-client-2: server 10.16.157.42:49152 has not responded in the last 42 seconds, disconnecting. >[2013-02-12 18:27:07.355947] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-master-client-3: server 10.16.157.45:49152 has not responded in the last 42 seconds, disconnecting. >[2013-02-12 18:27:07.356016] D [socket.c:184:__socket_rwv] 0-master-client-2: EOF from peer 10.16.157.42:49152 >[2013-02-12 18:27:07.356074] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.42:49152) >[2013-02-12 18:27:07.356095] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2013-02-12 18:27:07.356272] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-2: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:26:25.189476 (xid=0x9x) >[2013-02-12 18:27:07.356296] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) >[2013-02-12 18:27:07.356324] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-2: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-12 18:26:25.189484 (xid=0x10x) >[2013-02-12 18:27:07.356334] W [client-handshake.c:275:client_ping_cbk] 0-master-client-2: timer must have expired >[2013-02-12 18:27:07.356343] I [client.c:2098:client_rpc_notify] 0-master-client-2: disconnected >[2013-02-12 18:27:07.356354] D [fuse-bridge.c:4239:notify] 0-fuse: got event 8 on graph 0 >[2013-02-12 18:27:07.356363] D [socket.c:184:__socket_rwv] 0-master-client-3: EOF from peer 10.16.157.45:49152 >[2013-02-12 18:27:07.356371] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.45:49152) >[2013-02-12 18:27:07.356377] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2013-02-12 18:27:07.356403] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:26:25.189498 (xid=0x8x) >[2013-02-12 18:27:07.356415] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) >[2013-02-12 18:27:07.356534] D [name.c:149:client_fill_address_family] 0-master-client-3: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:27:07.358841] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.45 (port-24007) for hostname: 10.16.157.45 and port: 24007 >[2013-02-12 18:27:07.358886] I [socket.c:2315:socket_submit_request] 0-master-client-3: not connected (priv->connected = 0) >[2013-02-12 18:27:07.358897] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-master-client-3: failed to submit rpc-request (XID: 0x10x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (master-client-3) >[2013-02-12 18:27:07.358908] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) >[2013-02-12 18:27:07.358920] D [dht-common.c:434:dht_lookup_dir_cbk] 0-master-dht: lookup of / on master-replicate-1 returned error (Transport endpoint is not connected) >[2013-02-12 18:27:07.358932] D [client.c:234:client_submit_request] 0-master-client-3: rpc_clnt_submit failed >[2013-02-12 18:27:07.358964] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-3: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-12 18:26:25.189504 (xid=0x9x) >[2013-02-12 18:27:07.358978] W [client-handshake.c:275:client_ping_cbk] 0-master-client-3: timer must have expired >[2013-02-12 18:27:07.358986] I [client.c:2098:client_rpc_notify] 0-master-client-3: disconnected >[2013-02-12 18:27:07.358992] E [afr-common.c:3581:afr_notify] 0-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2013-02-12 18:27:07.359000] D [fuse-bridge.c:4239:notify] 0-fuse: got event 6 on graph 0 >[2013-02-12 18:27:07.359034] I [afr-common.c:1976:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2013-02-12 18:27:07.359071] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 240 3359 ] >[2013-02-12 18:27:07.359080] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 6342 7 ] >[2013-02-12 18:27:07.359088] D [afr-self-heal-common.c:829:afr_mark_sources] 0-master-replicate-0: Number of sources: 1 >[2013-02-12 18:27:07.359095] D [afr-self-heal-data.c:863:afr_lookup_select_read_child_by_txn_type] 0-master-replicate-0: returning read_child: 1 >[2013-02-12 18:27:07.359102] D [afr-common.c:1294:afr_lookup_select_read_child] 0-master-replicate-0: Source selected as 1 for / >[2013-02-12 18:27:07.359113] D [afr-common.c:1097:afr_lookup_build_response_params] 0-master-replicate-0: Building lookup response from 1 >[2013-02-12 18:27:07.359123] D [afr-common.c:1158:afr_lookup_set_self_heal_params_by_xattr] 0-master-replicate-0: entry self-heal is pending for /. >[2013-02-12 18:27:07.359130] D [afr-common.c:1158:afr_lookup_set_self_heal_params_by_xattr] 0-master-replicate-0: entry self-heal is pending for /. >[2013-02-12 18:27:07.359139] D [afr-common.c:1340:afr_launch_self_heal] 0-master-replicate-0: background entry self-heal triggered. path: /, reason: lookup detected pending operations >[2013-02-12 18:27:07.359156] D [afr-self-heal-metadata.c:63:afr_sh_metadata_done] 0-master-replicate-0: proceeding to entry check on / >[2013-02-12 18:27:07.359204] D [client.c:2051:client_rpc_notify] 0-master-client-3: got RPC_CLNT_CONNECT >[2013-02-12 18:27:07.359226] D [client-handshake.c:184:client_start_ping] 0-master-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:27:07.359919] D [afr-self-heal-entry.c:2384:afr_sh_post_nonblocking_entry_cbk] 0-master-replicate-0: Non Blocking entrylks done for /. Proceeding to FOP >[2013-02-12 18:27:07.359956] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up / on subvolume master-client-0 >[2013-02-12 18:27:07.360002] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up / on subvolume master-client-1 >[2013-02-12 18:27:07.361084] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 240 3359 ] >[2013-02-12 18:27:07.361109] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 6342 7 ] >[2013-02-12 18:27:07.361122] D [afr-self-heal-common.c:829:afr_mark_sources] 0-master-replicate-0: Number of sources: 1 >[2013-02-12 18:27:07.361134] I [afr-self-heal-entry.c:2335:afr_sh_entry_fix] 0-master-replicate-0: /: Performing conservative merge >[2013-02-12 18:27:07.361149] D [afr-self-heal-entry.c:2291:afr_sh_entry_sync_prepare] 0-master-replicate-0: no active sources for / found. merging all entries as a conservative decision >[2013-02-12 18:27:07.362417] D [afr-self-heal-entry.c:850:afr_sh_entry_expunge_all] 0-master-replicate-0: no active sources for / to expunge entries >[2013-02-12 18:27:07.362446] D [afr-self-heal-entry.c:2076:afr_sh_entry_impunge_subvol] 0-master-replicate-0: /: readdir from offset 0 >[2013-02-12 18:27:07.362852] D [afr-self-heal-entry.c:2050:afr_sh_entry_impunge_readdir_cbk] 0-master-replicate-0: readdir'ed 11 entries from master-client-0 >[2013-02-12 18:27:07.362911] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /8ff on subvolume master-client-0 >[2013-02-12 18:27:07.362961] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /8ff on subvolume master-client-1 >[2013-02-12 18:27:07.362995] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /940 on subvolume master-client-0 >[2013-02-12 18:27:07.363017] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /940 on subvolume master-client-1 >[2013-02-12 18:27:07.363044] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /970 on subvolume master-client-0 >[2013-02-12 18:27:07.363065] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /970 on subvolume master-client-1 >[2013-02-12 18:27:07.363096] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /99e on subvolume master-client-0 >[2013-02-12 18:27:07.363123] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /99e on subvolume master-client-1 >[2013-02-12 18:27:07.363153] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /9ca on subvolume master-client-0 >[2013-02-12 18:27:07.363186] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /9ca on subvolume master-client-1 >[2013-02-12 18:27:07.363218] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /a1c on subvolume master-client-0 >[2013-02-12 18:27:07.363242] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /a1c on subvolume master-client-1 >[2013-02-12 18:27:07.363271] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /a68 on subvolume master-client-0 >[2013-02-12 18:27:07.363300] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /a68 on subvolume master-client-1 >[2013-02-12 18:27:07.363339] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /perf-test on subvolume master-client-0 >[2013-02-12 18:27:07.363365] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /perf-test on subvolume master-client-1 >[2013-02-12 18:27:07.363394] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /linux2 on subvolume master-client-0 >[2013-02-12 18:27:07.363419] D [afr-self-heal-common.c:1906:afr_sh_common_lookup] 0-master-replicate-0: looking up /linux2 on subvolume master-client-1 >[2013-02-12 18:27:17.357425] D [name.c:149:client_fill_address_family] 0-master-client-2: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:27:17.359762] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.42 (port-24007) for hostname: 10.16.157.42 and port: 24007 >[2013-02-12 18:27:17.359963] D [client.c:2051:client_rpc_notify] 0-master-client-2: got RPC_CLNT_CONNECT >[2013-02-12 18:27:17.360048] D [client-handshake.c:184:client_start_ping] 0-master-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:27:49.364569] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) >[2013-02-12 18:28:31.370972] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-master-client-1: server 10.16.157.39:49152 has not responded in the last 42 seconds, disconnecting. >[2013-02-12 18:28:31.371141] D [socket.c:184:__socket_rwv] 0-master-client-1: EOF from peer 10.16.157.39:49152 >[2013-02-12 18:28:31.371198] D [socket.c:1512:__socket_proto_state_machine] 0-master-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.157.39:49152) >[2013-02-12 18:28:31.371219] D [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2013-02-12 18:28:31.371289] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.362985 (xid=0x13x) >[2013-02-12 18:28:31.371324] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /8ff (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371335] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /8ff on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371379] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363035 (xid=0x14x) >[2013-02-12 18:28:31.371391] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /940 (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371399] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /940 on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371424] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363085 (xid=0x15x) >[2013-02-12 18:28:31.371435] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /970 (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371442] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /970 on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371467] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363144 (xid=0x16x) >[2013-02-12 18:28:31.371478] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /99e (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371485] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /99e on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371510] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363206 (xid=0x17x) >[2013-02-12 18:28:31.371521] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /9ca (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371528] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /9ca on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371553] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363262 (xid=0x18x) >[2013-02-12 18:28:31.371567] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /a1c (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371575] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /a1c on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371600] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363327 (xid=0x19x) >[2013-02-12 18:28:31.371611] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /a68 (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371619] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /a68 on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371643] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363385 (xid=0x20x) >[2013-02-12 18:28:31.371654] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /perf-test (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371661] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /perf-test on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371686] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-12 18:27:07.363440 (xid=0x21x) >[2013-02-12 18:28:31.371697] W [client3_1-fops.c:2655:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected. Path: /linux2 (00000000-0000-0000-0000-000000000000) >[2013-02-12 18:28:31.371704] D [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-master-replicate-0: path /linux2 on subvolume master-client-1 => -1 (Transport endpoint is not connected) >[2013-02-12 18:28:31.371715] D [afr-self-heal-entry.c:2076:afr_sh_entry_impunge_subvol] 0-master-replicate-0: /: readdir from offset 2147483647 >[2013-02-12 18:28:31.371802] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3f06a0f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3f06a0f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3f06a0ef3e]))) 0-master-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-12 18:27:49.364662 (xid=0x22x) >[2013-02-12 18:28:31.371906] W [client-handshake.c:275:client_ping_cbk] 0-master-client-1: timer must have expired >[2013-02-12 18:28:31.371916] I [client.c:2098:client_rpc_notify] 0-master-client-1: disconnected >[2013-02-12 18:28:31.371929] D [fuse-bridge.c:4239:notify] 0-fuse: got event 8 on graph 0 >[2013-02-12 18:28:31.371950] D [afr-self-heal-entry.c:2076:afr_sh_entry_impunge_subvol] 0-master-replicate-0: /: readdir from offset 0 >[2013-02-12 18:28:31.371965] W [client3_1-fops.c:5652:client3_1_readdirp] 0-master-client-1: (00000000-0000-0000-0000-000000000001) remote_fd is -1. EBADFD >[2013-02-12 18:28:31.371974] I [afr-self-heal-entry.c:2030:afr_sh_entry_impunge_readdir_cbk] 0-master-replicate-0: readdir of / on subvolume master-client-1 failed (File descriptor in bad state) >[2013-02-12 18:28:31.371981] D [afr-lk-common.c:408:transaction_lk_op] 0-master-replicate-0: lk op is for a self heal >[2013-02-12 18:28:31.372009] D [client.c:180:client_submit_request] 0-master-client-1: connection in disconnected state >[2013-02-12 18:28:31.372025] W [client3_1-fops.c:1655:client3_1_entrylk_cbk] 0-master-client-1: remote operation failed: Transport endpoint is not connected >[2013-02-12 18:28:31.372076] E [afr-lk-common.c:703:afr_unlock_entrylk_cbk] 0-master-replicate-0: /: unlock failed on 1, reason: Transport endpoint is not connected >[2013-02-12 18:28:31.372101] E [afr-self-heal-common.c:2162:afr_self_heal_completion_cbk] 0-master-replicate-0: background entry self-heal failed on / >[2013-02-12 18:28:31.372118] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 240 3359 ] >[2013-02-12 18:28:31.372126] D [afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 0-master-replicate-0: pending_matrix: [ 6342 7 ] >[2013-02-12 18:28:31.372133] D [afr-self-heal-common.c:829:afr_mark_sources] 0-master-replicate-0: Number of sources: 1 >[2013-02-12 18:28:31.372140] D [afr-self-heal-data.c:863:afr_lookup_select_read_child_by_txn_type] 0-master-replicate-0: returning read_child: 1 >[2013-02-12 18:28:31.372147] D [afr-common.c:1294:afr_lookup_select_read_child] 0-master-replicate-0: Source selected as 1 for / >[2013-02-12 18:28:31.372156] D [afr-common.c:1097:afr_lookup_build_response_params] 0-master-replicate-0: Building lookup response from 1 >[2013-02-12 18:28:31.372170] I [dht-layout.c:598:dht_layout_normalize] 0-master-dht: found anomalies in /. holes=1 overlaps=0 >[2013-02-12 18:28:31.372218] D [dht-layout.c:614:dht_layout_normalize] (-->/usr/lib64/glusterfs/3.3.0.5rhs/xlator/cluster/replicate.so(afr_self_heal_completion_cbk+0x120) [0x7f962c6cff70] (-->/usr/lib64/glusterfs/3.3.0.5rhs/xlator/cluster/replicate.so(afr_self_heal_lookup_unwind+0x178) [0x7f962c6f1b68] (-->/usr/lib64/glusterfs/3.3.0.5rhs/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x5dc) [0x7f962c47e73c]))) 0-master-dht: path=/ err=Transport endpoint is not connected on subvol=master-replicate-1 >[2013-02-12 18:28:31.372230] D [dht-common.c:482:dht_lookup_dir_cbk] 0-master-dht: fixing assignment on / >[2013-02-12 18:28:31.372238] W [dht-selfheal.c:872:dht_selfheal_directory] 0-master-dht: 1 subvolumes down -- not fixing >[2013-02-12 18:28:31.372286] D [client3_1-fops.c:2806:client_fdctx_destroy] 0-master-client-0: sending releasedir on fd >[2013-02-12 18:28:41.372585] D [name.c:149:client_fill_address_family] 0-master-client-1: address-family not specified, guessing it to be inet/inet6 >[2013-02-12 18:28:41.375044] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.16.157.39 (port-24007) for hostname: 10.16.157.39 and port: 24007 >[2013-02-12 18:28:41.375343] D [client.c:2051:client_rpc_notify] 0-master-client-1: got RPC_CLNT_CONNECT >[2013-02-12 18:28:41.375398] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:29:13.379831] D [client-handshake.c:184:client_start_ping] 0-master-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) >[2013-02-12 18:29:26.729592] I [client-handshake.c:1614:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3.3.0.5rhs, Num (1298437), Version (330) >[2013-02-12 18:29:26.729669] D [client-handshake.c:184:client_start_ping] 0-master-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-02-12 18:29:26.731436] D [client-handshake.c:1385:client_setvolume_cbk] 0-master-client-1: clnt-lk-version = 1, server-lk-version = 0 >[2013-02-12 18:29:26.731475] I [client-handshake.c:1411:client_setvolume_cbk] 0-master-client-1: Connected to 10.16.157.39:49152, attached to remote volume '/exportdir/brick2'. >[2013-02-12 18:29:26.731489] I [client-handshake.c:1423:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2013-02-12 18:29:26.731501] I [client-handshake.c:1260:client_post_handshake] 0-master-client-1: 1 fds open - Delaying child_up until they are re-opened >[2013-02-12 18:29:26.731524] D [client-handshake.c:1122:protocol_client_reopendir] 0-master-client-1: attempting reopen on <gfid:00000000-0000-0000-0000-000000000001> >[2013-02-12 18:29:26.733431] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-master-client-1: reopendir on <gfid:00000000-0000-0000-0000-000000000001> succeeded (fd = 0) >[2013-02-12 18:29:26.733462] I [client-lk.c:601:decrement_reopen_fd_count] 0-master-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP >[2013-02-12 18:29:26.733486] D [client-handshake.c:489:client_set_lk_version] 0-master-client-1: Sending SET_LK_VERSION >[2013-02-12 18:29:26.733525] D [fuse-bridge.c:4239:notify] 0-fuse: got event 8 on graph 0 >[2013-02-12 18:29:26.733545] D [client3_1-fops.c:2806:client_fdctx_destroy] 0-master-client-1: sending releasedir on fd >[2013-02-12 18:29:26.734523] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 905010
:
688905
|
693286
| 696591