Attachment 611991 Details for Bug 856455 – mount-log

mount-log

mnt-test.log (text/x-log), 205.46 KB, created by Rachana Patel on 2012-09-12 05:25:45 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Rachana Patel

Created: 2012-09-12 05:25:45 UTC

Size: 205.46 KB

patch

obsolete

>[2012-09-11 10:26:55.550891] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhs
>[2012-09-11 10:26:55.769630] I [io-cache.c:1549:check_cache_size_ok] 0-test-quick-read: Max cache size is 2104406016
>[2012-09-11 10:26:55.769749] I [io-cache.c:1549:check_cache_size_ok] 0-test-io-cache: Max cache size is 2104406016
>[2012-09-11 10:26:55.774264] I [client.c:2142:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2012-09-11 10:26:55.776794] I [client.c:2142:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2012-09-11 10:26:55.778929] I [client.c:2142:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.35.63
>  4:     option remote-subvolume /home/test
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume test-client-1
>  9:     type protocol/client
> 10:     option remote-host 10.70.35.67
> 11:     option remote-subvolume /home/test
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume test-client-2
> 16:     type protocol/client
> 17:     option remote-host 10.70.35.65
> 18:     option remote-subvolume /home/test
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-client-0 test-client-1 test-client-2
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-md-cache
> 48:     type performance/md-cache
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes test-md-cache
> 57: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-11 10:26:55.781930] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-0: changing port to 24009 (from 0)
>[2012-09-11 10:26:55.782161] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-1: changing port to 24009 (from 0)
>[2012-09-11 10:26:55.782980] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-2: changing port to 24009 (from 0)
>[2012-09-11 10:26:59.560213] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 10:26:59.560614] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-0: Connected to 10.70.35.63:24009, attached to remote volume '/home/test'.
>[2012-09-11 10:26:59.758934] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 10:26:59.759495] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2012-09-11 10:26:59.759525] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 10:26:59.759598] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 10:26:59.759997] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 10:26:59.760023] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 10:26:59.760471] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 10:26:59.760533] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 10:26:59.760551] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 10:26:59.767255] I [fuse-bridge.c:4222:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-09-11 10:26:59.767402] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 10:26:59.767629] I [fuse-bridge.c:3405:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-09-11 11:04:56.141239] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 11:04:56.141372] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 11:05:06.827864] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 11:05:45.130788] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:05:45.131743] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:05:45.134145] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:05:45.136239] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (5f975038-4c2d-459d-9a1c-aebd47a70b6c)
>[2012-09-11 11:05:45.136770] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d3 (c384f6d7-591d-4bab-9f09-f9159b8d8d16)
>[2012-09-11 11:05:45.137329] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (4740d118-4088-43dc-bf09-8c89ef695c98)
>[2012-09-11 11:05:45.138024] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (5ea9a8fc-a68c-4732-a965-c241773c9a6c)
>[2012-09-11 11:05:45.138550] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (b22c15f0-6c56-4350-92fc-dd4653002977)
>[2012-09-11 11:05:45.141613] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (62811e4b-150d-4b06-87e6-70c77a6513d3)
>[2012-09-11 11:05:45.142387] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d8 (b3e9517b-38a1-4d49-8823-ee49500852d0)
>[2012-09-11 11:05:45.142942] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (76cffc85-a35b-4ca8-b13a-addf7fc7b97a)
>[2012-09-11 11:05:45.143446] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d10 (5b66e8cf-a87f-4d54-bb53-8dce16c85e9b)
>[2012-09-11 11:05:45.143857] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d11 (f38d9ff5-e0fd-4098-8468-876f9ccdf0a7)
>[2012-09-11 11:05:45.144285] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (46257513-3243-42f4-ae60-8e4e44e39a41)
>[2012-09-11 11:05:45.144705] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (714bd5fb-4394-4fa7-bcd9-dbaa8c7a0814)
>[2012-09-11 11:05:45.145139] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (b878c944-5052-4186-b574-ab7a7869efa1)
>[2012-09-11 11:05:45.145793] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (b470403e-1fc3-487e-8925-4e1a03f66314)
>[2012-09-11 11:05:45.146198] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (983b41c3-07a5-4cfc-b4e2-f5aa1001212a)
>[2012-09-11 11:05:45.146752] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (a6d089b2-7ef1-4cfd-b91d-7797806743ac)
>[2012-09-11 11:05:45.147100] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d18 (32104685-1bf1-434f-bb41-48057a63142b)
>[2012-09-11 11:05:45.147718] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d19 (cf4c5ebc-a56c-4153-9b1b-2386fc7ff82d)
>[2012-09-11 11:05:45.148030] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d20 (1a88c54e-b458-470e-a1ae-21f57d16614f)
>[2012-09-11 11:05:45.148591] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.150023] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:05:45.152190] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:05:45.152563] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:05:45.156587] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.157375] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.158962] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d10 (5b66e8cf-a87f-4d54-bb53-8dce16c85e9b)
>[2012-09-11 11:05:45.162996] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.163740] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.165506] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d11 (f38d9ff5-e0fd-4098-8468-876f9ccdf0a7)
>[2012-09-11 11:05:45.168137] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.168787] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.171864] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (46257513-3243-42f4-ae60-8e4e44e39a41)
>[2012-09-11 11:05:45.174667] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.175316] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.178528] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (714bd5fb-4394-4fa7-bcd9-dbaa8c7a0814)
>[2012-09-11 11:05:45.182536] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.183076] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.184842] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (b878c944-5052-4186-b574-ab7a7869efa1)
>[2012-09-11 11:05:45.188516] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.189181] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.190997] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (b470403e-1fc3-487e-8925-4e1a03f66314)
>[2012-09-11 11:05:45.193829] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.194533] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.197368] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (983b41c3-07a5-4cfc-b4e2-f5aa1001212a)
>[2012-09-11 11:05:45.200576] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.201580] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.204611] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (a6d089b2-7ef1-4cfd-b91d-7797806743ac)
>[2012-09-11 11:05:45.206636] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.207327] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.209851] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d18 (32104685-1bf1-434f-bb41-48057a63142b)
>[2012-09-11 11:05:45.212392] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.213264] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.216342] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d19 (cf4c5ebc-a56c-4153-9b1b-2386fc7ff82d)
>[2012-09-11 11:05:45.220170] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.220825] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.222330] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (5f975038-4c2d-459d-9a1c-aebd47a70b6c)
>[2012-09-11 11:05:45.226109] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.226827] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.228477] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d20 (1a88c54e-b458-470e-a1ae-21f57d16614f)
>[2012-09-11 11:05:45.230566] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.231305] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.233988] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d3 (c384f6d7-591d-4bab-9f09-f9159b8d8d16)
>[2012-09-11 11:05:45.237415] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.238002] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.239475] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (4740d118-4088-43dc-bf09-8c89ef695c98)
>[2012-09-11 11:05:45.243157] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.243696] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.245114] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (5ea9a8fc-a68c-4732-a965-c241773c9a6c)
>[2012-09-11 11:05:45.246745] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.247286] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.249287] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (b22c15f0-6c56-4350-92fc-dd4653002977)
>[2012-09-11 11:05:45.252769] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.253281] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.254890] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (62811e4b-150d-4b06-87e6-70c77a6513d3)
>[2012-09-11 11:05:45.256538] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.257086] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.259120] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d8 (b3e9517b-38a1-4d49-8823-ee49500852d0)
>[2012-09-11 11:05:45.262472] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.263117] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.264639] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (76cffc85-a35b-4ca8-b13a-addf7fc7b97a)
>[2012-09-11 11:05:45.267530] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:05:45.268335] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:16:48.167532] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:17:55.659436] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:17:55.662281] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:17:55.662331] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:17:55.663780] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:18:41.117348] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 11:18:41.117491] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 11:18:47.664785] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:18:47.665333] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /. holes=1 overlaps=0
>[2012-09-11 11:18:47.665359] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:47.665559] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:18:47.668037] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:18:48.678425] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:18:48.679049] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:18:48.681175] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:18:48.683837] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:18:48.683866] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:18:48.683915] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:18:48.683929] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:18:48.683943] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4123: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:18:48.684256] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:18:48.684275] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:18:48.684290] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4137: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:18:48.684385] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:18:48.684410] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:18:48.684443] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:18:48.684456] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:18:48.684470] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4138: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:18:48.684735] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:18:48.684754] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:18:48.684768] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4152: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:18:48.684860] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 22174717
>[2012-09-11 11:18:48.684877] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d11
>[2012-09-11 11:18:48.684907] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 22174717
>[2012-09-11 11:18:48.684919] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d11
>[2012-09-11 11:18:48.684932] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4153: LOOKUP() /d11 => -1 (Invalid argument)
>[2012-09-11 11:18:48.685147] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 22174717
>[2012-09-11 11:18:48.685164] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d11
>[2012-09-11 11:18:48.685178] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4167: LOOKUP() /d11 => -1 (Invalid argument)
>[2012-09-11 11:18:48.685725] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (46257513-3243-42f4-ae60-8e4e44e39a41)
>[2012-09-11 11:18:48.686132] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d12. holes=1 overlaps=0
>[2012-09-11 11:18:48.686167] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.686767] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (714bd5fb-4394-4fa7-bcd9-dbaa8c7a0814)
>[2012-09-11 11:18:48.687128] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d13. holes=1 overlaps=0
>[2012-09-11 11:18:48.687162] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.687737] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (b878c944-5052-4186-b574-ab7a7869efa1)
>[2012-09-11 11:18:48.688179] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d14. holes=1 overlaps=0
>[2012-09-11 11:18:48.688206] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.688761] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (b470403e-1fc3-487e-8925-4e1a03f66314)
>[2012-09-11 11:18:48.689085] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d15. holes=1 overlaps=0
>[2012-09-11 11:18:48.689109] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.689662] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (983b41c3-07a5-4cfc-b4e2-f5aa1001212a)
>[2012-09-11 11:18:48.690019] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d16. holes=1 overlaps=0
>[2012-09-11 11:18:48.690048] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.690478] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (a6d089b2-7ef1-4cfd-b91d-7797806743ac)
>[2012-09-11 11:18:48.690845] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d17. holes=1 overlaps=0
>[2012-09-11 11:18:48.690879] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.691038] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1323279326
>[2012-09-11 11:18:48.691060] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d18
>[2012-09-11 11:18:48.691097] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1323279326
>[2012-09-11 11:18:48.691113] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d18
>[2012-09-11 11:18:48.691129] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4174: LOOKUP() /d18 => -1 (Invalid argument)
>[2012-09-11 11:18:48.691398] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1323279326
>[2012-09-11 11:18:48.691430] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d18
>[2012-09-11 11:18:48.691447] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4188: LOOKUP() /d18 => -1 (Invalid argument)
>[2012-09-11 11:18:48.691558] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 390264492
>[2012-09-11 11:18:48.691577] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d19
>[2012-09-11 11:18:48.691612] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 390264492
>[2012-09-11 11:18:48.691626] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d19
>[2012-09-11 11:18:48.691642] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4189: LOOKUP() /d19 => -1 (Invalid argument)
>[2012-09-11 11:18:48.691928] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 390264492
>[2012-09-11 11:18:48.691950] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d19
>[2012-09-11 11:18:48.691966] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4203: LOOKUP() /d19 => -1 (Invalid argument)
>[2012-09-11 11:18:48.692428] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (5f975038-4c2d-459d-9a1c-aebd47a70b6c)
>[2012-09-11 11:18:48.692720] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d2. holes=1 overlaps=0
>[2012-09-11 11:18:48.692744] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.692849] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1248816474
>[2012-09-11 11:18:48.692865] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d20
>[2012-09-11 11:18:48.692891] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1248816474
>[2012-09-11 11:18:48.692902] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d20
>[2012-09-11 11:18:48.692913] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4205: LOOKUP() /d20 => -1 (Invalid argument)
>[2012-09-11 11:18:48.693101] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1248816474
>[2012-09-11 11:18:48.693116] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d20
>[2012-09-11 11:18:48.693127] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4219: LOOKUP() /d20 => -1 (Invalid argument)
>[2012-09-11 11:18:48.693206] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 722001417
>[2012-09-11 11:18:48.693220] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d3
>[2012-09-11 11:18:48.693244] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 722001417
>[2012-09-11 11:18:48.693254] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d3
>[2012-09-11 11:18:48.693265] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4220: LOOKUP() /d3 => -1 (Invalid argument)
>[2012-09-11 11:18:48.693472] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 722001417
>[2012-09-11 11:18:48.693486] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d3
>[2012-09-11 11:18:48.693502] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4234: LOOKUP() /d3 => -1 (Invalid argument)
>[2012-09-11 11:18:48.693937] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (4740d118-4088-43dc-bf09-8c89ef695c98)
>[2012-09-11 11:18:48.694221] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d4. holes=1 overlaps=0
>[2012-09-11 11:18:48.694245] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.694697] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (5ea9a8fc-a68c-4732-a965-c241773c9a6c)
>[2012-09-11 11:18:48.694979] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d5. holes=1 overlaps=0
>[2012-09-11 11:18:48.695005] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.695456] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (b22c15f0-6c56-4350-92fc-dd4653002977)
>[2012-09-11 11:18:48.695770] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d6. holes=1 overlaps=0
>[2012-09-11 11:18:48.695795] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.696252] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (62811e4b-150d-4b06-87e6-70c77a6513d3)
>[2012-09-11 11:18:48.696531] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d7. holes=1 overlaps=0
>[2012-09-11 11:18:48.696558] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:48.696721] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 565180887
>[2012-09-11 11:18:48.696745] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d8
>[2012-09-11 11:18:48.696784] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 565180887
>[2012-09-11 11:18:48.696799] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d8
>[2012-09-11 11:18:48.696815] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4239: LOOKUP() /d8 => -1 (Invalid argument)
>[2012-09-11 11:18:48.697087] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 565180887
>[2012-09-11 11:18:48.697109] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d8
>[2012-09-11 11:18:48.697125] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4253: LOOKUP() /d8 => -1 (Invalid argument)
>[2012-09-11 11:18:48.697503] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (76cffc85-a35b-4ca8-b13a-addf7fc7b97a)
>[2012-09-11 11:18:48.697791] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d9. holes=1 overlaps=0
>[2012-09-11 11:18:48.697816] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:18:49.902115] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:18:49.902843] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:18:49.902861] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:18:49.902886] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4257: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:18:54.670081] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 11:19:00.091169] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:19:00.091894] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:19:00.091921] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:19:00.091940] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4259: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:19:08.842148] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:19:08.843213] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:19:08.843240] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:19:08.843258] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4261: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:20:41.303176] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:20:41.303926] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:20:41.303952] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:20:41.303972] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4263: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:20:53.716159] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:20:53.716888] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:20:53.716906] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:20:53.716935] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4265: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:21:21.826415] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:21:21.826780] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:21:21.826797] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:21:21.827045] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:35:45.632181] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 11:35:45.632310] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 11:35:58.624186] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:35:58.625092] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:35:58.925971] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 11:36:33.144275] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.65:24009)
>[2012-09-11 11:36:33.144426] I [client.c:2090:client_rpc_notify] 0-test-client-2: disconnected
>[2012-09-11 11:36:39.601849] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:36:39.601918] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:36:39.602731] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:36:39.602780] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:36:39.603274] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0). Key: trusted.glusterfs.pathinfo
>[2012-09-11 11:36:39.603331] E [dht-common.c:1748:dht_vgetxattr_cbk] 0-test-dht: Subvolume test-client-2 returned -1 (Transport endpoint is not connected)
>[2012-09-11 11:36:39.603372] W [fuse-bridge.c:2841:fuse_xattr_cbk] 0-glusterfs-fuse: 4302: GETXATTR(trusted.glusterfs.pathinfo) /d1 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:36:43.977655] E [socket.c:1715:socket_connect_finish] 0-test-client-2: connection to 10.70.35.65:24009 failed (Connection refused)
>[2012-09-11 11:36:49.497100] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:36:49.497185] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:36:49.497841] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:36:49.497875] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:36:49.498350] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0). Key: trusted.glusterfs.pathinfo
>[2012-09-11 11:36:49.498376] E [dht-common.c:1748:dht_vgetxattr_cbk] 0-test-dht: Subvolume test-client-2 returned -1 (Transport endpoint is not connected)
>[2012-09-11 11:36:49.498394] W [fuse-bridge.c:2841:fuse_xattr_cbk] 0-glusterfs-fuse: 4305: GETXATTR(trusted.glusterfs.pathinfo) /d1 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:36:50.499758] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0). Key: trusted.glusterfs.pathinfo
>[2012-09-11 11:36:50.499805] E [dht-common.c:1748:dht_vgetxattr_cbk] 0-test-dht: Subvolume test-client-2 returned -1 (Transport endpoint is not connected)
>[2012-09-11 11:36:50.499824] W [fuse-bridge.c:2841:fuse_xattr_cbk] 0-glusterfs-fuse: 4306: GETXATTR(trusted.glusterfs.pathinfo) /d1 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:37:29.069511] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:37:29.072140] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:37:29.072198] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:37:29.072704] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 11:37:30.997758] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:37:30.998348] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /. holes=1 overlaps=0
>[2012-09-11 11:37:30.998415] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:37:30.998556] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:37:30.998578] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:37:30.998618] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:37:30.998633] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:37:30.998649] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4308: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:37:30.998810] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:37:30.998831] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:37:30.998847] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4310: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:37:31.776740] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:37:31.776775] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:37:31.776791] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4311: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:37:32.532095] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:37:32.533256] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 11:37:32.533283] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 11:37:32.533303] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 4313: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 11:37:35.065566] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:37:35.066147] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:37:35.066181] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:37:35.066627] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:38:19.510094] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 11:38:19.510222] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 11:38:20.404118] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:38:20.404909] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:38:21.511084] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:38:21.511836] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (529a2c16-329d-4b16-a7ba-a23b297c8bd0)
>[2012-09-11 11:38:33.078125] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 11:42:01.421124] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:42:16.746358] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:42:16.749008] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:42:16.749057] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:42:16.749510] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:43:18.426557] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 11:43:18.426704] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 11:43:31.754975] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 11:43:42.142159] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:43:42.142947] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (50a8350e-e575-4d9a-bc0d-cd7d780defe2)
>[2012-09-11 11:43:58.020847] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:43:58.022394] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /- (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:45:13.860312] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:45:13.891813] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:45:13.891848] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:45:13.892287] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:45:25.590752] I [fuse-bridge.c:4122:fuse_thread_proc] 0-fuse: unmounting /mnt/test
>[2012-09-11 11:45:25.592189] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37b66e76dd] (-->/lib64/libpthread.so.0() [0x37b6a07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405cfd]))) 0-: received signum (15), shutting down
>[2012-09-11 11:45:25.592223] I [fuse-bridge.c:4672:fini] 0-fuse: Unmounting '/mnt/test'.
>[2012-09-11 11:45:36.451448] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhs
>[2012-09-11 11:45:36.461720] I [io-cache.c:1549:check_cache_size_ok] 0-test-quick-read: Max cache size is 2104406016
>[2012-09-11 11:45:36.461784] I [io-cache.c:1549:check_cache_size_ok] 0-test-io-cache: Max cache size is 2104406016
>[2012-09-11 11:45:36.464354] I [client.c:2142:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2012-09-11 11:45:36.466603] I [client.c:2142:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2012-09-11 11:45:36.469939] I [client.c:2142:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.35.63
>  4:     option remote-subvolume /home/test
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume test-client-1
>  9:     type protocol/client
> 10:     option remote-host 10.70.35.67
> 11:     option remote-subvolume /home/test
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume test-client-2
> 16:     type protocol/client
> 17:     option remote-host 10.70.35.65
> 18:     option remote-subvolume /home/test
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-client-0 test-client-1 test-client-2
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-md-cache
> 48:     type performance/md-cache
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes test-md-cache
> 57: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-11 11:45:36.472632] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-0: changing port to 24009 (from 0)
>[2012-09-11 11:45:36.472697] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-1: changing port to 24009 (from 0)
>[2012-09-11 11:45:36.472897] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-2: changing port to 24009 (from 0)
>[2012-09-11 11:45:40.464137] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:45:40.464590] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-0: Connected to 10.70.35.63:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:45:40.464615] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:45:40.464880] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2012-09-11 11:45:40.467729] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:45:40.468155] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:45:40.468179] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:45:40.468443] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:45:40.471155] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:45:40.471508] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:45:40.471532] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:45:40.476375] I [fuse-bridge.c:4222:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-09-11 11:45:40.476492] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 11:45:40.476650] I [fuse-bridge.c:3405:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-09-11 11:45:57.089992] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 11:45:57.090122] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 11:46:04.277118] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:46:04.278291] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:46:04.278380] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:46:04.279044] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d1. holes=1 overlaps=0
>[2012-09-11 11:46:04.279073] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:46:10.476060] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 11:47:53.855928] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:47:53.857367] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /- (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:49:59.224148] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:49:59.225043] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d10 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:49:59.225115] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d10 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:49:59.225654] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d10. holes=1 overlaps=0
>[2012-09-11 11:49:59.225700] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:50:11.155457] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:50:11.156930] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /- (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.107060] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:51:13.109568] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:51:13.116016] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.119761] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (50a8350e-e575-4d9a-bc0d-cd7d780defe2)
>[2012-09-11 11:51:13.120397] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (50a8350e-e575-4d9a-bc0d-cd7d780defe2)
>[2012-09-11 11:51:13.122232] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.123513] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.125017] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.126349] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.127537] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.128978] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.130283] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.131628] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d1 (50a8350e-e575-4d9a-bc0d-cd7d780defe2)
>[2012-09-11 11:51:13.132334] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 52: RMDIR() /d1 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.133037] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d10 (f7b9997e-a06b-4e6c-957e-bcb6de3b457d)
>[2012-09-11 11:51:13.133486] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d10 (f7b9997e-a06b-4e6c-957e-bcb6de3b457d)
>[2012-09-11 11:51:13.134891] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.136903] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.138125] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.139374] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.140543] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.141727] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.142939] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d10 (f7b9997e-a06b-4e6c-957e-bcb6de3b457d)
>[2012-09-11 11:51:13.143599] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 83: RMDIR() /d10 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.144077] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d11 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.144148] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d11 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.144469] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d11. holes=1 overlaps=0
>[2012-09-11 11:51:13.144497] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.144656] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d11 (0e1848da-d305-4be3-8975-9e21e9a5f301)
>[2012-09-11 11:51:13.146136] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.148309] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.149812] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.150898] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.152091] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.153226] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.154643] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.155818] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.156873] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d11 (0e1848da-d305-4be3-8975-9e21e9a5f301)
>[2012-09-11 11:51:13.157480] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 122: RMDIR() /d11 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.158244] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.158544] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d12. holes=1 overlaps=0
>[2012-09-11 11:51:13.158574] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.158763] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (e84a0a02-8d57-438e-ac87-8549c9aadf90)
>[2012-09-11 11:51:13.160126] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.162180] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.163348] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.164447] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.165525] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.166872] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.167994] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.169102] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.170184] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (e84a0a02-8d57-438e-ac87-8549c9aadf90)
>[2012-09-11 11:51:13.170773] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 161: RMDIR() /d12 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.171539] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.171842] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d13. holes=1 overlaps=0
>[2012-09-11 11:51:13.171871] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.172037] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (4b8c3516-f5c6-4dff-9837-6985c499cc32)
>[2012-09-11 11:51:13.173318] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.175141] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.176261] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.177374] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.178457] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.179640] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.180726] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (4b8c3516-f5c6-4dff-9837-6985c499cc32)
>[2012-09-11 11:51:13.181333] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 192: RMDIR() /d13 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.182150] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.182496] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d14. holes=1 overlaps=0
>[2012-09-11 11:51:13.182524] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.182703] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (dbca356e-4b93-441d-9df5-986559394428)
>[2012-09-11 11:51:13.184123] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.186093] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.187397] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.188574] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.189629] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.190806] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.191897] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (dbca356e-4b93-441d-9df5-986559394428)
>[2012-09-11 11:51:13.192552] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 223: RMDIR() /d14 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.193302] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.193600] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d15. holes=1 overlaps=0
>[2012-09-11 11:51:13.193624] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.193803] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (514bdad8-1d19-4801-af19-2eaae356087d)
>[2012-09-11 11:51:13.195056] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.197002] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.198018] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.199110] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.200193] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.201251] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.202363] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.203607] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.204722] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (514bdad8-1d19-4801-af19-2eaae356087d)
>[2012-09-11 11:51:13.205302] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 262: RMDIR() /d15 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.206152] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.206487] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d16. holes=1 overlaps=0
>[2012-09-11 11:51:13.206515] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.206695] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (0e4c9211-8abd-45e3-b78e-65e62ddbc3e2)
>[2012-09-11 11:51:13.208090] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.210129] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.211200] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.212342] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.213369] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.214391] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.215511] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.216571] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.217638] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (0e4c9211-8abd-45e3-b78e-65e62ddbc3e2)
>[2012-09-11 11:51:13.218304] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 301: RMDIR() /d16 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.219146] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.219477] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d17. holes=1 overlaps=0
>[2012-09-11 11:51:13.219505] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.219662] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (80b10112-3e54-45c9-ac05-3c94500125ce)
>[2012-09-11 11:51:13.221001] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.223445] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.224652] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.225872] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.227105] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.228304] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.229611] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (80b10112-3e54-45c9-ac05-3c94500125ce)
>[2012-09-11 11:51:13.230314] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 332: RMDIR() /d17 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.230596] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d18 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.230915] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d18 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.231223] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d18. holes=1 overlaps=0
>[2012-09-11 11:51:13.231255] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.231440] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d18 (05176c71-ab2e-4f29-ace8-0b31b57c1f5b)
>[2012-09-11 11:51:13.232790] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.234827] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.235904] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.236943] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.237946] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.239091] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.240239] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.241329] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.242512] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d18 (05176c71-ab2e-4f29-ace8-0b31b57c1f5b)
>[2012-09-11 11:51:13.243133] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 371: RMDIR() /d18 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.243569] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d19 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.243638] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d19 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.244009] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d19. holes=1 overlaps=0
>[2012-09-11 11:51:13.244057] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.244255] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d19 (f9f0f3f7-5708-48da-827f-658d36d08d67)
>[2012-09-11 11:51:13.245586] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.247517] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.248651] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.249722] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.250747] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.251931] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.253067] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d19 (f9f0f3f7-5708-48da-827f-658d36d08d67)
>[2012-09-11 11:51:13.253690] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 402: RMDIR() /d19 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.254412] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.254792] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d2. holes=1 overlaps=0
>[2012-09-11 11:51:13.254822] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.254990] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (8f6890d5-6462-4157-8cc7-8fe3d704359f)
>[2012-09-11 11:51:13.256319] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.258165] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.259257] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.260344] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.261432] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.262617] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.263690] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (8f6890d5-6462-4157-8cc7-8fe3d704359f)
>[2012-09-11 11:51:13.264276] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 433: RMDIR() /d2 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.264794] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d20 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.264865] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d20 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.265156] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d20. holes=1 overlaps=0
>[2012-09-11 11:51:13.265195] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.265385] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d20 (56bde979-6cec-4e58-82d1-0af9a9e51300)
>[2012-09-11 11:51:13.266615] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.268697] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.269721] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.270927] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.271980] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.272975] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.274032] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d20 (56bde979-6cec-4e58-82d1-0af9a9e51300)
>[2012-09-11 11:51:13.274600] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 464: RMDIR() /d20 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.275045] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d3 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.275115] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d3 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.275434] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d3. holes=1 overlaps=0
>[2012-09-11 11:51:13.275462] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.275619] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d3 (2292379a-6561-40d9-a304-cf6a0d4f0b57)
>[2012-09-11 11:51:13.276932] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.278872] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.279910] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.280920] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.281916] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.283065] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.284206] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d3 (2292379a-6561-40d9-a304-cf6a0d4f0b57)
>[2012-09-11 11:51:13.284830] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 495: RMDIR() /d3 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.285578] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.285889] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d4. holes=1 overlaps=0
>[2012-09-11 11:51:13.285930] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.286098] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (270734ae-6db5-4512-8a39-c5cc066b651d)
>[2012-09-11 11:51:13.287548] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.289489] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.290652] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.291754] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.292845] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.294003] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.295167] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (270734ae-6db5-4512-8a39-c5cc066b651d)
>[2012-09-11 11:51:13.295814] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 526: RMDIR() /d4 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.301106] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.301438] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d5. holes=1 overlaps=0
>[2012-09-11 11:51:13.301467] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.301622] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (612e4fd4-f269-493e-b05b-182c34d46e57)
>[2012-09-11 11:51:13.302882] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.305051] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.306230] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.307307] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.308350] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.309337] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.310315] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (612e4fd4-f269-493e-b05b-182c34d46e57)
>[2012-09-11 11:51:13.310915] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 557: RMDIR() /d5 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.311560] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.311896] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d6. holes=1 overlaps=0
>[2012-09-11 11:51:13.311926] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.312081] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (9b226c64-fbdf-4141-954a-36121e1e654b)
>[2012-09-11 11:51:13.313407] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.315191] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.316271] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.317325] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.318403] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.319475] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.320625] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (9b226c64-fbdf-4141-954a-36121e1e654b)
>[2012-09-11 11:51:13.321235] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 588: RMDIR() /d6 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.321910] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.322217] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d7. holes=1 overlaps=0
>[2012-09-11 11:51:13.322252] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.322447] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (a76bf0c2-a2df-44c4-8f28-0fbf2956a9a8)
>[2012-09-11 11:51:13.323711] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.325773] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.326898] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.327995] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.329124] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.330253] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.331420] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (a76bf0c2-a2df-44c4-8f28-0fbf2956a9a8)
>[2012-09-11 11:51:13.332056] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 619: RMDIR() /d7 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.332539] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d8 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.332606] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d8 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.332878] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d8. holes=1 overlaps=0
>[2012-09-11 11:51:13.332910] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.333108] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d8 (f8e5c939-4e1e-4657-a840-eba908e0148d)
>[2012-09-11 11:51:13.334486] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.336405] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.337429] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.338462] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.339547] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.340707] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.341730] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d8 (f8e5c939-4e1e-4657-a840-eba908e0148d)
>[2012-09-11 11:51:13.342361] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 650: RMDIR() /d8 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:13.343184] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:51:13.343493] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d9. holes=1 overlaps=0
>[2012-09-11 11:51:13.343522] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:51:13.343707] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (d615763b-1728-48b7-ae54-2c65018733b0)
>[2012-09-11 11:51:13.345119] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.347166] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.348315] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.349325] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.350358] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.351551] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.352605] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.353626] W [client3_1-fops.c:474:client3_1_stat_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 11:51:13.354639] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (d615763b-1728-48b7-ae54-2c65018733b0)
>[2012-09-11 11:51:13.355235] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 689: RMDIR() /d9 => -1 (Transport endpoint is not connected)
>[2012-09-11 11:51:31.834525] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:51:31.837188] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:51:31.837244] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:51:31.837615] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:51:45.352009] I [fuse-bridge.c:4122:fuse_thread_proc] 0-fuse: unmounting /mnt/test
>[2012-09-11 11:51:45.352433] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37b66e76dd] (-->/lib64/libpthread.so.0() [0x37b6a07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405cfd]))) 0-: received signum (15), shutting down
>[2012-09-11 11:51:45.352463] I [fuse-bridge.c:4672:fini] 0-fuse: Unmounting '/mnt/test'.
>[2012-09-11 11:52:54.268068] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhs
>[2012-09-11 11:52:54.278163] I [io-cache.c:1549:check_cache_size_ok] 0-test-quick-read: Max cache size is 2104406016
>[2012-09-11 11:52:54.278224] I [io-cache.c:1549:check_cache_size_ok] 0-test-io-cache: Max cache size is 2104406016
>[2012-09-11 11:52:54.279687] I [client.c:2142:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2012-09-11 11:52:54.282412] I [client.c:2142:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2012-09-11 11:52:54.284523] I [client.c:2142:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.35.63
>  4:     option remote-subvolume /home/test
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume test-client-1
>  9:     type protocol/client
> 10:     option remote-host 10.70.35.67
> 11:     option remote-subvolume /home/test
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume test-client-2
> 16:     type protocol/client
> 17:     option remote-host 10.70.35.65
> 18:     option remote-subvolume /home/test
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-client-0 test-client-1 test-client-2
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-md-cache
> 48:     type performance/md-cache
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes test-md-cache
> 57: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-11 11:52:54.287632] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-0: changing port to 24009 (from 0)
>[2012-09-11 11:52:54.287757] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-1: changing port to 24009 (from 0)
>[2012-09-11 11:52:54.288163] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-2: changing port to 24009 (from 0)
>[2012-09-11 11:52:58.280874] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:52:58.281431] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-0: Connected to 10.70.35.63:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:52:58.281456] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:52:58.281792] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2012-09-11 11:52:58.284355] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:52:58.284807] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:52:58.284884] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:52:58.285206] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:52:58.287219] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:52:58.287558] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:52:58.287574] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:52:58.289874] I [fuse-bridge.c:4222:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-09-11 11:52:58.289991] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 11:52:58.290142] I [fuse-bridge.c:3405:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-09-11 11:55:21.927283] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 11:55:21.927538] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 11:55:32.308384] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 11:56:02.343839] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 11:56:02.346162] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 11:56:02.346216] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 11:56:02.346515] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 11:56:24.656087] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 11:56:24.656220] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 11:56:38.350147] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 11:56:52.102743] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:56:52.103340] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /. holes=1 overlaps=0
>[2012-09-11 11:56:52.103380] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 11:56:52.104548] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:56:52.104574] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:56:52.104619] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:56:52.104635] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:56:52.104651] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3423: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:56:52.105053] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:56:52.105075] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:56:52.105092] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3437: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:58:47.612599] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:58:47.614817] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /- (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:59:35.653117] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:59:35.653888] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:59:35.653915] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:59:35.653935] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3441: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:59:36.408015] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:59:36.408059] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:59:36.408081] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3442: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:59:37.084049] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:59:37.085098] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:59:37.085128] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:59:37.085147] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3444: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:59:37.760041] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:59:37.760084] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:59:37.760106] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3445: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:59:38.324086] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:59:38.325028] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 11:59:38.325054] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 11:59:38.325074] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3447: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 11:59:44.404971] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 11:59:44.405612] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1004596370
>[2012-09-11 11:59:44.405632] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /ifcon
>[2012-09-11 11:59:44.405646] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3449: LOOKUP() /ifcon => -1 (Invalid argument)
>[2012-09-11 11:59:44.406898] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /ifconfig (00000000-0000-0000-0000-000000000000)
>[2012-09-11 11:59:44.407943] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /ifconfig (00000000-0000-0000-0000-000000000000)
>[2012-09-11 12:00:30.362107] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:00:30.363002] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:00:30.363047] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:00:30.363065] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3453: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:00:51.459063] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:00:51.459771] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:00:51.459798] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:00:51.459817] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3455: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:00:59.674434] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:01:00.041786] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:01:00.041827] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:01:00.041845] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3457: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:02:02.471102] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:02.472329] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:02:02.479950] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:02:02.479985] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3459: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:02:03.821969] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:03.822709] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:03.825180] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 12:02:03.825210] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 12:02:03.825252] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 12:02:03.825267] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 12:02:03.825284] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3464: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 12:02:03.825668] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 12:02:03.825714] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 12:02:03.825731] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3478: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 12:02:03.826424] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (744f1b13-d2b0-4c44-9a3c-f6415d46e9d4)
>[2012-09-11 12:02:03.826782] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d2. holes=1 overlaps=0
>[2012-09-11 12:02:03.826812] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.826901] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 722001417
>[2012-09-11 12:02:03.826915] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d3
>[2012-09-11 12:02:03.826940] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 722001417
>[2012-09-11 12:02:03.826977] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d3
>[2012-09-11 12:02:03.826990] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3480: LOOKUP() /d3 => -1 (Invalid argument)
>[2012-09-11 12:02:03.827246] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 722001417
>[2012-09-11 12:02:03.827260] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d3
>[2012-09-11 12:02:03.827302] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3494: LOOKUP() /d3 => -1 (Invalid argument)
>[2012-09-11 12:02:03.827722] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (263d1005-54fa-425f-b85b-e927e435feaa)
>[2012-09-11 12:02:03.828042] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d4. holes=1 overlaps=0
>[2012-09-11 12:02:03.828058] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.828554] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (19bc8d2f-d0dd-4c53-beb4-f48ae90776d3)
>[2012-09-11 12:02:03.828839] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d5. holes=1 overlaps=0
>[2012-09-11 12:02:03.828858] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.829210] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (ac532f3b-aca6-4489-a96d-19d7eff7901c)
>[2012-09-11 12:02:03.829725] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d6. holes=1 overlaps=0
>[2012-09-11 12:02:03.829748] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.830179] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (c0e9cd45-5930-49f0-a589-de1045cf055f)
>[2012-09-11 12:02:03.830422] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d7. holes=1 overlaps=0
>[2012-09-11 12:02:03.830438] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.830521] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 565180887
>[2012-09-11 12:02:03.830535] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d8
>[2012-09-11 12:02:03.830561] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 565180887
>[2012-09-11 12:02:03.830571] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d8
>[2012-09-11 12:02:03.830582] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3499: LOOKUP() /d8 => -1 (Invalid argument)
>[2012-09-11 12:02:03.830790] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 565180887
>[2012-09-11 12:02:03.830805] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d8
>[2012-09-11 12:02:03.830816] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3513: LOOKUP() /d8 => -1 (Invalid argument)
>[2012-09-11 12:02:03.831244] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (4f0f0577-6e6b-427b-87a2-820409188b9d)
>[2012-09-11 12:02:03.831477] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d9. holes=1 overlaps=0
>[2012-09-11 12:02:03.831493] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.831582] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:02:03.831596] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:02:03.831616] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3515: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:02:03.831762] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 22174717
>[2012-09-11 12:02:03.831778] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d11
>[2012-09-11 12:02:03.831803] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 22174717
>[2012-09-11 12:02:03.831814] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d11
>[2012-09-11 12:02:03.831824] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3516: LOOKUP() /d11 => -1 (Invalid argument)
>[2012-09-11 12:02:03.832014] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 22174717
>[2012-09-11 12:02:03.832028] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d11
>[2012-09-11 12:02:03.832039] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3530: LOOKUP() /d11 => -1 (Invalid argument)
>[2012-09-11 12:02:03.832395] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (ccd6a09d-cdc5-49ea-9b07-dc4bd26a16ab)
>[2012-09-11 12:02:03.832654] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d12. holes=1 overlaps=0
>[2012-09-11 12:02:03.832740] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.833049] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (7b07b833-e98a-4dba-b16f-582ada6d404d)
>[2012-09-11 12:02:03.833231] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d13. holes=1 overlaps=0
>[2012-09-11 12:02:03.833247] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.833528] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (c0ceaa11-708f-4e11-9a22-61bf284ff9e3)
>[2012-09-11 12:02:03.833708] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d14. holes=1 overlaps=0
>[2012-09-11 12:02:03.833724] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.833993] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (68672270-e513-465b-b4b9-47abb0cd776a)
>[2012-09-11 12:02:03.834157] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d15. holes=1 overlaps=0
>[2012-09-11 12:02:03.834172] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.834438] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (2a5b325e-fd2b-4631-aa08-c33d3aabb9b0)
>[2012-09-11 12:02:03.834602] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d16. holes=1 overlaps=0
>[2012-09-11 12:02:03.834617] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.834998] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (bf017741-a56f-4fa7-85a7-ec27f21a4ead)
>[2012-09-11 12:02:03.835203] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /d17. holes=1 overlaps=0
>[2012-09-11 12:02:03.835219] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 12:02:03.835303] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1323279326
>[2012-09-11 12:02:03.835317] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d18
>[2012-09-11 12:02:03.835342] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1323279326
>[2012-09-11 12:02:03.835362] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d18
>[2012-09-11 12:02:03.835374] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3537: LOOKUP() /d18 => -1 (Invalid argument)
>[2012-09-11 12:02:03.835593] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1323279326
>[2012-09-11 12:02:03.835608] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d18
>[2012-09-11 12:02:03.835619] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3551: LOOKUP() /d18 => -1 (Invalid argument)
>[2012-09-11 12:02:03.835760] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 390264492
>[2012-09-11 12:02:03.835776] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d19
>[2012-09-11 12:02:03.835801] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 390264492
>[2012-09-11 12:02:03.835811] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d19
>[2012-09-11 12:02:03.835821] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3552: LOOKUP() /d19 => -1 (Invalid argument)
>[2012-09-11 12:02:03.836002] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 390264492
>[2012-09-11 12:02:03.836016] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d19
>[2012-09-11 12:02:03.836027] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3566: LOOKUP() /d19 => -1 (Invalid argument)
>[2012-09-11 12:02:03.836107] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1248816474
>[2012-09-11 12:02:03.836121] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d20
>[2012-09-11 12:02:03.836174] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1248816474
>[2012-09-11 12:02:03.836185] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d20
>[2012-09-11 12:02:03.836195] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3567: LOOKUP() /d20 => -1 (Invalid argument)
>[2012-09-11 12:02:03.836373] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1248816474
>[2012-09-11 12:02:03.836386] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d20
>[2012-09-11 12:02:03.836403] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3581: LOOKUP() /d20 => -1 (Invalid argument)
>[2012-09-11 12:02:03.836767] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:02:03.838703] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:18.178376] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:18.179167] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:18.181641] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 12:02:18.181693] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 12:02:18.181718] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3592: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 12:02:18.182039] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (744f1b13-d2b0-4c44-9a3c-f6415d46e9d4)
>[2012-09-11 12:02:18.182638] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 722001417
>[2012-09-11 12:02:18.182660] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d3
>[2012-09-11 12:02:18.182753] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3594: LOOKUP() /d3 => -1 (Invalid argument)
>[2012-09-11 12:02:18.182987] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (263d1005-54fa-425f-b85b-e927e435feaa)
>[2012-09-11 12:02:18.183600] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (19bc8d2f-d0dd-4c53-beb4-f48ae90776d3)
>[2012-09-11 12:02:18.184219] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (ac532f3b-aca6-4489-a96d-19d7eff7901c)
>[2012-09-11 12:02:18.184800] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (c0e9cd45-5930-49f0-a589-de1045cf055f)
>[2012-09-11 12:02:18.185364] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 565180887
>[2012-09-11 12:02:18.185396] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d8
>[2012-09-11 12:02:18.185415] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3599: LOOKUP() /d8 => -1 (Invalid argument)
>[2012-09-11 12:02:18.185590] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (4f0f0577-6e6b-427b-87a2-820409188b9d)
>[2012-09-11 12:02:18.186076] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:02:18.186100] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:02:18.186117] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3601: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:02:18.186277] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 22174717
>[2012-09-11 12:02:18.186298] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d11
>[2012-09-11 12:02:18.186314] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3602: LOOKUP() /d11 => -1 (Invalid argument)
>[2012-09-11 12:02:18.186492] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (ccd6a09d-cdc5-49ea-9b07-dc4bd26a16ab)
>[2012-09-11 12:02:18.187058] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (7b07b833-e98a-4dba-b16f-582ada6d404d)
>[2012-09-11 12:02:18.187539] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (c0ceaa11-708f-4e11-9a22-61bf284ff9e3)
>[2012-09-11 12:02:18.188062] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (68672270-e513-465b-b4b9-47abb0cd776a)
>[2012-09-11 12:02:18.188573] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (2a5b325e-fd2b-4631-aa08-c33d3aabb9b0)
>[2012-09-11 12:02:18.189050] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (bf017741-a56f-4fa7-85a7-ec27f21a4ead)
>[2012-09-11 12:02:18.189500] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1323279326
>[2012-09-11 12:02:18.189526] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d18
>[2012-09-11 12:02:18.189545] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3609: LOOKUP() /d18 => -1 (Invalid argument)
>[2012-09-11 12:02:18.189770] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 390264492
>[2012-09-11 12:02:18.189805] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d19
>[2012-09-11 12:02:18.189837] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3610: LOOKUP() /d19 => -1 (Invalid argument)
>[2012-09-11 12:02:18.189976] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1248816474
>[2012-09-11 12:02:18.189997] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d20
>[2012-09-11 12:02:18.190014] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3611: LOOKUP() /d20 => -1 (Invalid argument)
>[2012-09-11 12:02:18.190439] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:02:18.192224] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:22.475623] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:22.476302] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:02:22.476320] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:02:22.476333] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3619: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:02:27.197563] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:27.198760] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:27.201020] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 12:02:27.201040] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 12:02:27.201053] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3624: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 12:02:27.201228] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d2 (744f1b13-d2b0-4c44-9a3c-f6415d46e9d4)
>[2012-09-11 12:02:27.202243] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 722001417
>[2012-09-11 12:02:27.202260] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d3
>[2012-09-11 12:02:27.202272] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3627: LOOKUP() /d3 => -1 (Invalid argument)
>[2012-09-11 12:02:27.202387] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d4 (263d1005-54fa-425f-b85b-e927e435feaa)
>[2012-09-11 12:02:27.202891] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d5 (19bc8d2f-d0dd-4c53-beb4-f48ae90776d3)
>[2012-09-11 12:02:27.203331] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (ac532f3b-aca6-4489-a96d-19d7eff7901c)
>[2012-09-11 12:02:27.203813] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d7 (c0e9cd45-5930-49f0-a589-de1045cf055f)
>[2012-09-11 12:02:27.204206] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 565180887
>[2012-09-11 12:02:27.204222] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d8
>[2012-09-11 12:02:27.204233] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3636: LOOKUP() /d8 => -1 (Invalid argument)
>[2012-09-11 12:02:27.204344] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d9 (4f0f0577-6e6b-427b-87a2-820409188b9d)
>[2012-09-11 12:02:27.204824] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:02:27.204841] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:02:27.204853] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3639: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:02:27.204943] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 22174717
>[2012-09-11 12:02:27.204956] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d11
>[2012-09-11 12:02:27.204967] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3640: LOOKUP() /d11 => -1 (Invalid argument)
>[2012-09-11 12:02:27.205074] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d12 (ccd6a09d-cdc5-49ea-9b07-dc4bd26a16ab)
>[2012-09-11 12:02:27.205621] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d13 (7b07b833-e98a-4dba-b16f-582ada6d404d)
>[2012-09-11 12:02:27.206195] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d14 (c0ceaa11-708f-4e11-9a22-61bf284ff9e3)
>[2012-09-11 12:02:27.206774] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d15 (68672270-e513-465b-b4b9-47abb0cd776a)
>[2012-09-11 12:02:27.207196] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d16 (2a5b325e-fd2b-4631-aa08-c33d3aabb9b0)
>[2012-09-11 12:02:27.207609] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d17 (bf017741-a56f-4fa7-85a7-ec27f21a4ead)
>[2012-09-11 12:02:27.208041] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1323279326
>[2012-09-11 12:02:27.208061] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d18
>[2012-09-11 12:02:27.208073] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3653: LOOKUP() /d18 => -1 (Invalid argument)
>[2012-09-11 12:02:27.208167] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 390264492
>[2012-09-11 12:02:27.208181] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d19
>[2012-09-11 12:02:27.208192] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3654: LOOKUP() /d19 => -1 (Invalid argument)
>[2012-09-11 12:02:27.208279] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1248816474
>[2012-09-11 12:02:27.208293] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d20
>[2012-09-11 12:02:27.208304] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3655: LOOKUP() /d20 => -1 (Invalid argument)
>[2012-09-11 12:02:27.208642] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:02:27.210389] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:47.219399] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:02:59.005234] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:02:59.005284] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:02:59.005306] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3664: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:03:01.226368] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:06:13.107297] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 1262629412
>[2012-09-11 12:06:13.107351] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d10
>[2012-09-11 12:06:13.107386] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3666: LOOKUP() /d10 => -1 (Invalid argument)
>[2012-09-11 12:06:26.664494] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:06:26.667575] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (ac532f3b-aca6-4489-a96d-19d7eff7901c)
>[2012-09-11 12:07:16.977204] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:07:16.977931] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d6 (ac532f3b-aca6-4489-a96d-19d7eff7901c)
>[2012-09-11 12:09:45.638242] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:10:24.816504] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:12:24.903062] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:13:07.872658] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-test-client-0: server 10.70.35.63:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-11 12:13:07.875844] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-test-client-2: server 10.70.35.65:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-11 12:14:33.929923] I [socket.c:2315:socket_submit_request] 0-test-client-0: not connected (priv->connected = 255)
>[2012-09-11 12:14:33.930209] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x1803x Program: GlusterFS 3.1, ProgVers: 330, Proc: 14) to rpc-transport (test-client-0)
>[2012-09-11 12:14:33.930231] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:14:33.930248] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:14:33.930262] I [socket.c:2315:socket_submit_request] 0-test-client-2: not connected (priv->connected = 255)
>[2012-09-11 12:14:33.930271] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x1641x Program: GlusterFS 3.1, ProgVers: 330, Proc: 14) to rpc-transport (test-client-2)
>[2012-09-11 12:14:33.930281] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:14:33.930291] W [fuse-bridge.c:2525:fuse_statfs_cbk] 0-glusterfs-fuse: 3676: ERR => -1 (Transport endpoint is not connected)
>[2012-09-11 12:15:22.535319] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x1804x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 12:15:22.535395] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:15:22.535426] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:15:22.535451] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x1642x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 12:15:22.535522] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:15:22.535548] W [fuse-bridge.c:513:fuse_attr_cbk] 0-glusterfs-fuse: 3677: LOOKUP() / => -1 (Transport endpoint is not connected)
>[2012-09-11 12:15:51.229153] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x1805x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 12:15:51.229439] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:15:51.229460] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:15:51.229477] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x1643x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 12:15:51.229489] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:16:07.937987] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x1806x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 12:16:07.938037] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:16:07.938056] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:16:07.938073] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x1644x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 12:16:07.938085] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:16:39.346243] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x1807x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 12:16:39.346503] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:16:39.346523] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:16:39.346540] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x1645x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 12:16:39.346553] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 12:16:48.927469] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x1808x Program: GlusterFS 3.1, ProgVers: 330, Proc: 14) to rpc-transport (test-client-0)
>[2012-09-11 12:16:48.927519] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:16:48.927534] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:16:48.927547] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x1646x Program: GlusterFS 3.1, ProgVers: 330, Proc: 14) to rpc-transport (test-client-2)
>[2012-09-11 12:16:48.927569] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected
>[2012-09-11 12:16:48.927579] W [fuse-bridge.c:2525:fuse_statfs_cbk] 0-glusterfs-fuse: 3681: ERR => -1 (Transport endpoint is not connected)
>[2012-09-11 12:19:32.278395] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhs
>[2012-09-11 12:19:32.289093] I [io-cache.c:1549:check_cache_size_ok] 0-test-quick-read: Max cache size is 2104406016
>[2012-09-11 12:19:32.289159] I [io-cache.c:1549:check_cache_size_ok] 0-test-io-cache: Max cache size is 2104406016
>[2012-09-11 12:19:32.290881] I [client.c:2142:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2012-09-11 12:19:32.293096] I [client.c:2142:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2012-09-11 12:19:32.295284] I [client.c:2142:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.35.63
>  4:     option remote-subvolume /home/test
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume test-client-1
>  9:     type protocol/client
> 10:     option remote-host 10.70.35.67
> 11:     option remote-subvolume /home/test
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume test-client-2
> 16:     type protocol/client
> 17:     option remote-host 10.70.35.65
> 18:     option remote-subvolume /home/test
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-client-0 test-client-1 test-client-2
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-md-cache
> 48:     type performance/md-cache
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes test-md-cache
> 57: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-11 12:19:32.298425] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-1: changing port to 24009 (from 0)
>[2012-09-11 12:19:32.298468] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-0: changing port to 24009 (from 0)
>[2012-09-11 12:19:32.298504] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-2: changing port to 24009 (from 0)
>[2012-09-11 12:19:36.289079] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:19:36.289716] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:19:36.289742] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:19:36.290075] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 12:19:36.292312] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:19:36.292729] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-0: Connected to 10.70.35.63:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:19:36.292767] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:19:36.293016] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2012-09-11 12:19:36.295859] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:19:36.296990] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:19:36.297024] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:19:36.300608] I [fuse-bridge.c:4222:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-09-11 12:19:36.300766] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 12:19:36.301400] I [fuse-bridge.c:3405:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-09-11 12:22:27.400182] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhs
>[2012-09-11 12:22:27.411302] I [io-cache.c:1549:check_cache_size_ok] 0-test-quick-read: Max cache size is 2104406016
>[2012-09-11 12:22:27.411393] I [io-cache.c:1549:check_cache_size_ok] 0-test-io-cache: Max cache size is 2104406016
>[2012-09-11 12:22:27.413640] I [client.c:2142:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2012-09-11 12:22:27.416379] I [client.c:2142:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2012-09-11 12:22:27.418464] I [client.c:2142:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.35.63
>  4:     option remote-subvolume /home/test
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume test-client-1
>  9:     type protocol/client
> 10:     option remote-host 10.70.35.67
> 11:     option remote-subvolume /home/test
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume test-client-2
> 16:     type protocol/client
> 17:     option remote-host 10.70.35.65
> 18:     option remote-subvolume /home/test
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-client-0 test-client-1 test-client-2
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-md-cache
> 48:     type performance/md-cache
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes test-md-cache
> 57: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-11 12:22:27.421259] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-0: changing port to 24009 (from 0)
>[2012-09-11 12:22:27.421362] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-2: changing port to 24009 (from 0)
>[2012-09-11 12:22:27.421413] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-1: changing port to 24009 (from 0)
>[2012-09-11 12:22:31.411070] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:22:31.411599] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-0: Connected to 10.70.35.63:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:22:31.411621] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:22:31.411986] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2012-09-11 12:22:31.413461] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:22:31.413747] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:22:31.413768] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:22:31.413997] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 12:22:31.416129] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:22:31.416447] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:22:31.416474] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:22:31.420008] I [fuse-bridge.c:4222:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-09-11 12:22:31.420134] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 12:22:31.420287] I [fuse-bridge.c:3405:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-09-11 12:24:49.301421] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhs
>[2012-09-11 12:24:49.308619] I [io-cache.c:1549:check_cache_size_ok] 0-test-quick-read: Max cache size is 2104406016
>[2012-09-11 12:24:49.308694] I [io-cache.c:1549:check_cache_size_ok] 0-test-io-cache: Max cache size is 2104406016
>[2012-09-11 12:24:49.310778] I [client.c:2142:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2012-09-11 12:24:49.312938] I [client.c:2142:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2012-09-11 12:24:49.315645] I [client.c:2142:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.35.63
>  4:     option remote-subvolume /home/test
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume test-client-1
>  9:     type protocol/client
> 10:     option remote-host 10.70.35.67
> 11:     option remote-subvolume /home/test
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume test-client-2
> 16:     type protocol/client
> 17:     option remote-host 10.70.35.65
> 18:     option remote-subvolume /home/test
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-client-0 test-client-1 test-client-2
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-md-cache
> 48:     type performance/md-cache
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes test-md-cache
> 57: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-11 12:24:49.318521] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-1: changing port to 24009 (from 0)
>[2012-09-11 12:24:49.318585] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-0: changing port to 24009 (from 0)
>[2012-09-11 12:24:49.318629] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-2: changing port to 24009 (from 0)
>[2012-09-11 12:24:53.314021] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:24:53.314537] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:24:53.314562] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:24:53.314880] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 12:24:53.316813] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:24:53.317056] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-0: Connected to 10.70.35.63:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:24:53.317071] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:24:53.317222] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2012-09-11 12:24:53.319185] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:24:53.319509] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:24:53.319525] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:24:53.322220] I [fuse-bridge.c:4222:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-09-11 12:24:53.322344] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 12:24:53.322511] I [fuse-bridge.c:3405:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-09-11 12:25:42.545575] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhs
>[2012-09-11 12:25:42.552931] I [io-cache.c:1549:check_cache_size_ok] 0-test-quick-read: Max cache size is 2104406016
>[2012-09-11 12:25:42.552990] I [io-cache.c:1549:check_cache_size_ok] 0-test-io-cache: Max cache size is 2104406016
>[2012-09-11 12:25:42.554877] I [client.c:2142:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2012-09-11 12:25:42.557582] I [client.c:2142:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2012-09-11 12:25:42.559637] I [client.c:2142:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.35.63
>  4:     option remote-subvolume /home/test
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume test-client-1
>  9:     type protocol/client
> 10:     option remote-host 10.70.35.67
> 11:     option remote-subvolume /home/test
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume test-client-2
> 16:     type protocol/client
> 17:     option remote-host 10.70.35.65
> 18:     option remote-subvolume /home/test
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-client-0 test-client-1 test-client-2
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-md-cache
> 48:     type performance/md-cache
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes test-md-cache
> 57: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-11 12:25:42.562729] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-2: changing port to 24009 (from 0)
>[2012-09-11 12:25:42.562831] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-1: changing port to 24009 (from 0)
>[2012-09-11 12:25:42.562895] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-0: changing port to 24009 (from 0)
>[2012-09-11 12:25:46.558083] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:25:46.558615] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:25:46.558638] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:25:46.558933] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 12:25:46.561076] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:25:46.561454] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:25:46.561477] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:25:46.561720] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 12:25:46.564432] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 12:25:46.564862] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-0: Connected to 10.70.35.63:24009, attached to remote volume '/home/test'.
>[2012-09-11 12:25:46.564893] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 12:25:46.567462] I [fuse-bridge.c:4222:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-09-11 12:25:46.567581] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2012-09-11 12:25:46.567753] I [fuse-bridge.c:3405:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-09-11 13:11:04.844069] W [socket.c:195:__socket_rwv] 0-test-client-1: writev failed (Broken pipe)
>[2012-09-11 13:11:46.861043] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-test-client-0: server 10.70.35.63:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-11 13:11:46.863789] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-test-client-1: server 10.70.35.67:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-11 13:11:46.863872] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-test-client-2: server 10.70.35.65:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-11 13:41:07.060537] E [rpc-clnt.c:208:call_bail] 0-test-client-2: bailing out frame type(GlusterFS Handshake) op(PING(3)) xid = 0x80x sent = 2012-09-11 13:11:04.844238. timeout = 1800
>[2012-09-11 13:41:07.060610] W [client-handshake.c:275:client_ping_cbk] 0-test-client-2: timer must have expired
>[2012-09-11 13:41:07.060628] E [rpc-clnt.c:208:call_bail] 0-test-client-2: bailing out frame type(GlusterFS 3.1) op(LOOKUP(27)) xid = 0x79x sent = 2012-09-11 13:11:04.844230. timeout = 1800
>[2012-09-11 13:41:07.060647] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.060691] E [rpc-clnt.c:208:call_bail] 0-test-client-1: bailing out frame type(GlusterFS Handshake) op(PING(3)) xid = 0x78x sent = 2012-09-11 13:11:04.844207. timeout = 1800
>[2012-09-11 13:41:07.060706] W [client-handshake.c:275:client_ping_cbk] 0-test-client-1: timer must have expired
>[2012-09-11 13:41:07.060720] E [rpc-clnt.c:208:call_bail] 0-test-client-1: bailing out frame type(GlusterFS 3.1) op(LOOKUP(27)) xid = 0x77x sent = 2012-09-11 13:11:04.844203. timeout = 1800
>[2012-09-11 13:41:07.060733] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.060750] E [rpc-clnt.c:208:call_bail] 0-test-client-0: bailing out frame type(GlusterFS Handshake) op(PING(3)) xid = 0x79x sent = 2012-09-11 13:11:04.844049. timeout = 1800
>[2012-09-11 13:41:07.060761] W [client-handshake.c:275:client_ping_cbk] 0-test-client-0: timer must have expired
>[2012-09-11 13:41:07.060774] E [rpc-clnt.c:208:call_bail] 0-test-client-0: bailing out frame type(GlusterFS 3.1) op(LOOKUP(27)) xid = 0x78x sent = 2012-09-11 13:11:04.844038. timeout = 1800
>[2012-09-11 13:41:07.060787] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.060811] W [fuse-bridge.c:513:fuse_attr_cbk] 0-glusterfs-fuse: 91: LOOKUP() / => -1 (Transport endpoint is not connected)
>[2012-09-11 13:41:07.060999] I [socket.c:2315:socket_submit_request] 0-test-client-0: not connected (priv->connected = 255)
>[2012-09-11 13:41:07.061018] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x81x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 13:41:07.061034] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.061054] I [socket.c:2315:socket_submit_request] 0-test-client-1: not connected (priv->connected = 255)
>[2012-09-11 13:41:07.061067] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x80x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 13:41:07.061081] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.061100] I [socket.c:2315:socket_submit_request] 0-test-client-2: not connected (priv->connected = 255)
>[2012-09-11 13:41:07.061112] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x82x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 13:41:07.061125] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.062830] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x82x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 13:41:07.062850] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.062866] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x81x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 13:41:07.062878] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.062892] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x83x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 13:41:07.062903] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.062985] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x83x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 13:41:07.063001] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.063015] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x82x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 13:41:07.063027] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:07.063041] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x84x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 13:41:07.063052] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065241] E [rpc-clnt.c:208:call_bail] 0-test-client-2: bailing out frame type(GlusterFS 3.1) op(LOOKUP(27)) xid = 0x81x sent = 2012-09-11 13:11:37.599520. timeout = 1800
>[2012-09-11 13:41:47.065303] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065325] E [rpc-clnt.c:208:call_bail] 0-test-client-1: bailing out frame type(GlusterFS 3.1) op(LOOKUP(27)) xid = 0x79x sent = 2012-09-11 13:11:37.599488. timeout = 1800
>[2012-09-11 13:41:47.065337] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065350] E [rpc-clnt.c:208:call_bail] 0-test-client-0: bailing out frame type(GlusterFS 3.1) op(LOOKUP(27)) xid = 0x80x sent = 2012-09-11 13:11:37.599480. timeout = 1800
>[2012-09-11 13:41:47.065360] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065546] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x84x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 13:41:47.065565] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065582] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x83x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 13:41:47.065604] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065620] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x85x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 13:41:47.065632] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065735] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x85x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 13:41:47.065753] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065768] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x84x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 13:41:47.065780] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 13:41:47.065794] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x86x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 13:41:47.065805] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:08:53.704318] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x86x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:08:53.704386] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:08:53.704418] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x85x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:08:53.704437] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:08:53.704460] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x87x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:08:53.704477] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:07.890864] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x87x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:09:07.890920] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:07.890950] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x86x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:09:07.890969] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:07.890991] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x88x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:09:07.891020] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.897998] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x88x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:09:14.898058] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.898087] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x87x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:09:14.898106] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.898127] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x89x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:09:14.898144] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.898265] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x89x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:09:14.898289] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.898310] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x88x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:09:14.898327] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.898363] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x90x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:09:14.898381] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.898492] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x90x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:09:14.898515] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.898536] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x89x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:09:14.898553] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:14.898572] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x91x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:09:14.898588] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:25.067263] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x91x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:09:25.067362] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:25.067395] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x90x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:09:25.067415] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:25.067437] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x92x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:09:25.067454] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212011] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x92x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:09:27.212067] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212096] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x91x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:09:27.212115] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212136] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x93x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:09:27.212153] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212274] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x93x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:09:27.212297] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212319] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x92x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:09:27.212359] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212385] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x94x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:09:27.212402] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212515] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-0: failed to submit rpc-request (XID: 0x94x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-0)
>[2012-09-11 14:09:27.212538] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212559] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-1: failed to submit rpc-request (XID: 0x93x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-1)
>[2012-09-11 14:09:27.212585] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:27.212606] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-test-client-2: failed to submit rpc-request (XID: 0x95x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (test-client-2)
>[2012-09-11 14:09:27.212622] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:09:40.104699] I [fuse-bridge.c:4122:fuse_thread_proc] 0-fuse: unmounting /mnt/test
>[2012-09-11 14:09:40.105604] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37b66e76dd] (-->/lib64/libpthread.so.0() [0x37b6a07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405cfd]))) 0-: received signum (15), shutting down
>[2012-09-11 14:09:40.105624] I [fuse-bridge.c:4672:fini] 0-fuse: Unmounting '/mnt/test'.
>[2012-09-11 14:10:29.350145] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhs
>[2012-09-11 14:10:29.363305] I [io-cache.c:1549:check_cache_size_ok] 0-test-quick-read: Max cache size is 2104406016
>[2012-09-11 14:10:29.363373] I [io-cache.c:1549:check_cache_size_ok] 0-test-io-cache: Max cache size is 2104406016
>[2012-09-11 14:10:29.364871] I [client.c:2142:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2012-09-11 14:10:29.367867] I [client.c:2142:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2012-09-11 14:10:29.369914] I [client.c:2142:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.35.63
>  4:     option remote-subvolume /home/test
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume test-client-1
>  9:     type protocol/client
> 10:     option remote-host 10.70.35.67
> 11:     option remote-subvolume /home/test
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume test-client-2
> 16:     type protocol/client
> 17:     option remote-host 10.70.35.65
> 18:     option remote-subvolume /home/test
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-client-0 test-client-1 test-client-2
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-md-cache
> 48:     type performance/md-cache
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes test-md-cache
> 57: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-11 14:10:29.373021] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-1: changing port to 24009 (from 0)
>[2012-09-11 14:10:29.373093] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-0: changing port to 24009 (from 0)
>[2012-09-11 14:10:29.373303] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-test-client-2: changing port to 24009 (from 0)
>[2012-09-11 14:10:33.366018] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 14:10:33.366768] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 14:10:33.366795] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 14:10:33.367052] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 14:10:33.368086] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 14:10:33.368321] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-0: Connected to 10.70.35.63:24009, attached to remote volume '/home/test'.
>[2012-09-11 14:10:33.368343] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 14:10:33.368494] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2012-09-11 14:10:33.370400] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 14:10:33.370604] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-2: Connected to 10.70.35.65:24009, attached to remote volume '/home/test'.
>[2012-09-11 14:10:33.370620] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 14:10:33.373079] I [fuse-bridge.c:4222:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-09-11 14:10:33.373199] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2012-09-11 14:10:33.373364] I [fuse-bridge.c:3405:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-09-11 14:13:32.220023] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 14:13:32.220164] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 14:13:42.394551] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 14:14:15.347129] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:14:15.347829] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /d10 (9aa20d77-74ca-4785-b5df-183cedbc84bb)
>[2012-09-11 14:14:30.444081] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 14:14:30.446727] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 14:14:30.446774] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 14:14:30.447117] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 14:14:56.585967] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 14:14:56.586091] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 14:15:04.564710] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:15:04.567044] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /. holes=1 overlaps=0
>[2012-09-11 14:15:04.567090] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing
>[2012-09-11 14:15:04.567262] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 14:15:04.567284] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 14:15:04.567323] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 14:15:04.567361] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 14:15:04.567384] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3115: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 14:15:04.567655] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 14:15:04.567698] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 14:15:04.567716] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3127: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 14:15:10.450020] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 14:16:19.551314] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:16:19.551989] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 14:16:19.552016] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 14:16:19.552034] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3129: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 14:19:36.296097] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:19:36.297399] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /hist (00000000-0000-0000-0000-000000000000)
>[2012-09-11 14:19:36.299524] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /history (00000000-0000-0000-0000-000000000000)
>[2012-09-11 14:19:36.300504] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: /history (00000000-0000-0000-0000-000000000000)
>[2012-09-11 14:20:20.853423] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-11 14:20:20.854782] W [dht-layout.c:186:dht_layout_search] 0-test-dht: no subvolume for hash (value) = 184435781
>[2012-09-11 14:20:20.854812] E [dht-common.c:1372:dht_lookup] 0-test-dht: Failed to get hashed subvol for /d1
>[2012-09-11 14:20:20.854833] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3135: LOOKUP() /d1 => -1 (Invalid argument)
>[2012-09-11 14:35:17.709618] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 14:35:17.710083] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 14:35:17.710107] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 14:35:17.710392] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 14:41:33.168869] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 14:41:33.169008] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 14:41:43.754098] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 14:41:43.756749] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 14:41:43.756807] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 14:41:43.757317] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 14:42:19.394797] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 14:42:19.394938] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 14:42:32.761101] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 14:43:11.794569] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 14:43:11.795084] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 14:43:11.795108] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 14:43:11.795519] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 14:43:36.457641] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 14:43:36.457852] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 14:43:49.802040] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 14:44:13.821611] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 14:44:13.822183] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 14:44:13.822205] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 14:44:13.822589] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 14:44:50.873223] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 14:44:50.873361] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 14:45:04.830050] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 15:16:09.770513] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 15:16:09.770972] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 15:16:09.770996] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 15:16:09.771294] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 15:20:18.416104] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.67:24009)
>[2012-09-11 15:20:18.416209] I [client.c:2090:client_rpc_notify] 0-test-client-1: disconnected
>[2012-09-11 15:20:31.803035] E [socket.c:1715:socket_connect_finish] 0-test-client-1: connection to 10.70.35.67:24009 failed (Connection refused)
>[2012-09-11 18:31:47.034541] I [client-handshake.c:1636:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3.0rhs, Num (1298437), Version (330)
>[2012-09-11 18:31:47.037049] I [client-handshake.c:1433:client_setvolume_cbk] 0-test-client-1: Connected to 10.70.35.67:24009, attached to remote volume '/home/test'.
>[2012-09-11 18:31:47.037109] I [client-handshake.c:1445:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-11 18:31:47.037533] I [client-handshake.c:453:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2012-09-11 18:33:17.176141] W [socket.c:1512:__socket_proto_state_machine] 0-test-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.35.65:24009)
>[2012-09-11 18:33:17.176260] I [client.c:2090:client_rpc_notify] 0-test-client-2: disconnected
>[2012-09-11 18:33:28.048310] E [socket.c:1715:socket_connect_finish] 0-test-client-2: connection to 10.70.35.65:24009 failed (Connection refused)
>[2012-09-12 03:43:01.796859] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-test-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-12 03:43:01.799175] I [dht-layout.c:593:dht_layout_normalize] 0-test-dht: found anomalies in /. holes=1 overlaps=0
>[2012-09-12 03:43:01.799212] W [dht-selfheal.c:875:dht_selfheal_directory] 0-test-dht: 1 subvolumes down -- not fixing

Actions: View

Attachments on bug 856455: 611991