Attachment 589744 Details for Bug 829168 – glustershd log

glustershd log

glustershd.log (text/x-log), 85.11 KB, created by Shwetha Panduranga on 2012-06-06 06:41:37 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Shwetha Panduranga

Created: 2012-06-06 06:41:37 UTC

Size: 85.11 KB

patch

obsolete

>[2012-06-05 03:49:49.178292] I [glusterfsd.c:1666:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.3.0qa45
>[2012-06-05 03:49:49.229770] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-2: adding option 'node-uuid' for volume 'vol-replicate-2' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d'
>[2012-06-05 03:49:49.229811] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-1: adding option 'node-uuid' for volume 'vol-replicate-1' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d'
>[2012-06-05 03:49:49.229823] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-0: adding option 'node-uuid' for volume 'vol-replicate-0' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d'
>[2012-06-05 03:49:49.249452] I [client.c:2142:notify] 0-vol-client-0: parent translators are ready, attempting connect on transport
>[2012-06-05 03:49:49.252189] I [client.c:2142:notify] 0-vol-client-1: parent translators are ready, attempting connect on transport
>[2012-06-05 03:49:49.255765] I [client.c:2142:notify] 0-vol-client-2: parent translators are ready, attempting connect on transport
>[2012-06-05 03:49:49.258421] I [client.c:2142:notify] 0-vol-client-3: parent translators are ready, attempting connect on transport
>[2012-06-05 03:49:49.261057] I [client.c:2142:notify] 0-vol-client-4: parent translators are ready, attempting connect on transport
>[2012-06-05 03:49:49.264776] I [client.c:2142:notify] 0-vol-client-5: parent translators are ready, attempting connect on transport
>[2012-06-05 03:49:49.268040] I [client.c:2142:notify] 0-vol-client-6: parent translators are ready, attempting connect on transport
>[2012-06-05 03:49:49.274895] I [client.c:2142:notify] 0-vol-client-7: parent translators are ready, attempting connect on transport
>[2012-06-05 03:49:49.281484] I [client.c:2142:notify] 0-vol-client-8: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume vol-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.16.159.184
>  4:     option remote-subvolume /export_b1/dir1
>  5:     option transport-type tcp
>  6:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
>  7:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
>  8: end-volume
>  9: 
> 10: volume vol-client-1
> 11:     type protocol/client
> 12:     option remote-host 10.16.159.188
> 13:     option remote-subvolume /export_b1/dir1
> 14:     option transport-type tcp
> 15:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 16:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 17: end-volume
> 18: 
> 19: volume vol-client-2
> 20:     type protocol/client
> 21:     option remote-host 10.16.159.196
> 22:     option remote-subvolume /export_b1/dir1
> 23:     option transport-type tcp
> 24:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 25:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 26: end-volume
> 27: 
> 28: volume vol-client-3
> 29:     type protocol/client
> 30:     option remote-host 10.16.159.184
> 31:     option remote-subvolume /export_c1/dir1
> 32:     option transport-type tcp
> 33:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 34:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 35: end-volume
> 36: 
> 37: volume vol-client-4
> 38:     type protocol/client
> 39:     option remote-host 10.16.159.188
> 40:     option remote-subvolume /export_c1/dir1
> 41:     option transport-type tcp
> 42:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 43:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 44: end-volume
> 45: 
> 46: volume vol-client-5
> 47:     type protocol/client
> 48:     option remote-host 10.16.159.196
> 49:     option remote-subvolume /export_c1/dir1
> 50:     option transport-type tcp
> 51:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 52:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 53: end-volume
> 54: 
> 55: volume vol-client-6
> 56:     type protocol/client
> 57:     option remote-host 10.16.159.184
> 58:     option remote-subvolume /export_d1/dir1
> 59:     option transport-type tcp
> 60:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 61:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 62: end-volume
> 63: 
> 64: volume vol-client-7
> 65:     type protocol/client
> 66:     option remote-host 10.16.159.188
> 67:     option remote-subvolume /export_d1/dir1
> 68:     option transport-type tcp
> 69:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 70:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 71: end-volume
> 72: 
> 73: volume vol-client-8
> 74:     type protocol/client
> 75:     option remote-host 10.16.159.196
> 76:     option remote-subvolume /export_d1/dir1
> 77:     option transport-type tcp
> 78:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 79:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 80: end-volume
> 81: 
> 82: volume vol-replicate-0
> 83:     type cluster/replicate
> 84:     option background-self-heal-count 0
> 85:     option metadata-self-heal on
> 86:     option data-self-heal on
> 87:     option entry-self-heal on
> 88:     option self-heal-daemon on
> 89:     option iam-self-heal-daemon yes
> 90:     subvolumes vol-client-0 vol-client-1 vol-client-2
> 91: end-volume
> 92: 
> 93: volume vol-replicate-1
> 94:     type cluster/replicate
> 95:     option background-self-heal-count 0
> 96:     option metadata-self-heal on
> 97:     option data-self-heal on
> 98:     option entry-self-heal on
> 99:     option self-heal-daemon on
>100:     option iam-self-heal-daemon yes
>101:     subvolumes vol-client-3 vol-client-4 vol-client-5
>102: end-volume
>103: 
>104: volume vol-replicate-2
>105:     type cluster/replicate
>106:     option background-self-heal-count 0
>107:     option metadata-self-heal on
>108:     option data-self-heal on
>109:     option entry-self-heal on
>110:     option self-heal-daemon on
>111:     option iam-self-heal-daemon yes
>112:     subvolumes vol-client-6 vol-client-7 vol-client-8
>113: end-volume
>114: 
>115: volume glustershd
>116:     type debug/io-stats
>117:     subvolumes vol-replicate-0 vol-replicate-1 vol-replicate-2
>118: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-05 03:49:49.287840] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-0: changing port to 24009 (from 0)
>[2012-06-05 03:49:49.287981] E [client-handshake.c:1717:client_query_portmap_cbk] 0-vol-client-1: failed to get the port number for remote subvolume
>[2012-06-05 03:49:49.288091] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-3: changing port to 24010 (from 0)
>[2012-06-05 03:49:49.288217] E [client-handshake.c:1717:client_query_portmap_cbk] 0-vol-client-4: failed to get the port number for remote subvolume
>[2012-06-05 03:49:49.288893] I [client.c:2090:client_rpc_notify] 0-vol-client-1: disconnected
>[2012-06-05 03:49:49.289038] I [client.c:2090:client_rpc_notify] 0-vol-client-4: disconnected
>[2012-06-05 03:49:49.289125] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-6: changing port to 24011 (from 0)
>[2012-06-05 03:49:49.289251] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-2: changing port to 24009 (from 0)
>[2012-06-05 03:49:49.289412] E [client-handshake.c:1717:client_query_portmap_cbk] 0-vol-client-7: failed to get the port number for remote subvolume
>[2012-06-05 03:49:49.289740] I [client.c:2090:client_rpc_notify] 0-vol-client-7: disconnected
>[2012-06-05 03:49:49.289832] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-5: changing port to 24010 (from 0)
>[2012-06-05 03:49:49.289991] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-8: changing port to 24011 (from 0)
>[2012-06-05 03:49:53.021102] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-06-05 03:49:53.021180] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-06-05 03:49:53.192702] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:53.193499] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-0: Connected to 10.16.159.184:24009, attached to remote volume '/export_b1/dir1'.
>[2012-06-05 03:49:53.193562] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:53.193705] I [afr-common.c:3627:afr_notify] 0-vol-replicate-0: Subvolume 'vol-client-0' came back up; going online.
>[2012-06-05 03:49:53.194080] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-0: Server lk version = 1
>[2012-06-05 03:49:53.201215] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-1: changing port to 24009 (from 0)
>[2012-06-05 03:49:53.210293] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:53.211016] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-3: Connected to 10.16.159.184:24010, attached to remote volume '/export_c1/dir1'.
>[2012-06-05 03:49:53.211074] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:53.211219] I [afr-common.c:3627:afr_notify] 0-vol-replicate-1: Subvolume 'vol-client-3' came back up; going online.
>[2012-06-05 03:49:53.211583] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-3: Server lk version = 1
>[2012-06-05 03:49:53.217711] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-4: changing port to 24010 (from 0)
>[2012-06-05 03:49:53.225556] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-6: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:53.226209] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-6: Connected to 10.16.159.184:24011, attached to remote volume '/export_d1/dir1'.
>[2012-06-05 03:49:53.226257] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:53.226404] I [afr-common.c:3627:afr_notify] 0-vol-replicate-2: Subvolume 'vol-client-6' came back up; going online.
>[2012-06-05 03:49:53.229414] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-6: Server lk version = 1
>[2012-06-05 03:49:53.233557] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:53.234423] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-2: Connected to 10.16.159.196:24009, attached to remote volume '/export_b1/dir1'.
>[2012-06-05 03:49:53.234473] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:53.235378] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-2: Server lk version = 1
>[2012-06-05 03:49:53.240898] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-7: changing port to 24011 (from 0)
>[2012-06-05 03:49:53.248910] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-5: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:53.249626] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-5: Connected to 10.16.159.196:24010, attached to remote volume '/export_c1/dir1'.
>[2012-06-05 03:49:53.249861] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:53.251687] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-5: Server lk version = 1
>[2012-06-05 03:49:53.257387] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-8: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:53.258008] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-8: Connected to 10.16.159.196:24011, attached to remote volume '/export_d1/dir1'.
>[2012-06-05 03:49:53.258044] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:53.260705] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-8: Server lk version = 1
>[2012-06-05 03:49:56.262358] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:56.262875] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-1: Connected to 10.16.159.188:24009, attached to remote volume '/export_b1/dir1'.
>[2012-06-05 03:49:56.262911] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:56.263573] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-1: Server lk version = 1
>[2012-06-05 03:49:56.267918] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-4: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:56.268425] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-4: Connected to 10.16.159.188:24010, attached to remote volume '/export_c1/dir1'.
>[2012-06-05 03:49:56.268459] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:56.269091] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-4: Server lk version = 1
>[2012-06-05 03:49:56.274539] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-7: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-05 03:49:56.274998] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-7: Connected to 10.16.159.188:24011, attached to remote volume '/export_d1/dir1'.
>[2012-06-05 03:49:56.275033] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-05 03:49:56.275730] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-7: Server lk version = 1
>[2012-06-05 03:50:04.679071] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.188:24009)
>[2012-06-05 03:50:04.679190] I [client.c:2090:client_rpc_notify] 0-vol-client-1: disconnected
>[2012-06-05 03:50:12.908907] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-4: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.188:24010)
>[2012-06-05 03:50:12.909038] I [client.c:2090:client_rpc_notify] 0-vol-client-4: disconnected
>[2012-06-05 03:50:15.283671] E [socket.c:1715:socket_connect_finish] 0-vol-client-1: connection to 10.16.159.188:24009 failed (Connection refused)
>[2012-06-05 03:50:21.717155] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-7: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.188:24011)
>[2012-06-05 03:50:21.717263] I [client.c:2090:client_rpc_notify] 0-vol-client-7: disconnected
>[2012-06-05 03:50:23.299485] E [socket.c:1715:socket_connect_finish] 0-vol-client-4: connection to 10.16.159.188:24010 failed (Connection refused)
>[2012-06-05 03:50:32.320252] E [socket.c:1715:socket_connect_finish] 0-vol-client-7: connection to 10.16.159.188:24011 failed (Connection refused)
>[2012-06-05 03:50:41.749353] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.196:24009)
>[2012-06-05 03:50:41.749503] I [client.c:2090:client_rpc_notify] 0-vol-client-2: disconnected
>[2012-06-05 03:50:47.653867] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-5: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.196:24010)
>[2012-06-05 03:50:47.653979] I [client.c:2090:client_rpc_notify] 0-vol-client-5: disconnected
>[2012-06-05 03:50:52.421570] E [socket.c:1715:socket_connect_finish] 0-vol-client-2: connection to 10.16.159.196:24009 failed (Connection refused)
>[2012-06-05 03:50:52.675259] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-8: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.196:24011)
>[2012-06-05 03:50:52.675451] I [client.c:2090:client_rpc_notify] 0-vol-client-8: disconnected
>[2012-06-05 03:50:58.460806] E [socket.c:1715:socket_connect_finish] 0-vol-client-5: connection to 10.16.159.196:24010 failed (Connection refused)
>[2012-06-05 03:51:03.501953] E [socket.c:1715:socket_connect_finish] 0-vol-client-8: connection to 10.16.159.196:24011 failed (Connection refused)
>[2012-06-05 03:59:57.036775] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 03:59:57.039657] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 03:59:57.039675] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 04:09:57.729266] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 04:09:57.729405] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 04:09:57.729869] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 04:19:58.244458] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 04:19:58.244643] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 04:19:58.245888] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 04:29:58.919633] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 04:29:58.920034] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 04:29:58.925338] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 04:39:59.467531] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 04:39:59.470316] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 04:39:59.470687] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 04:50:00.088726] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 04:50:00.091523] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 04:50:00.091904] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 05:00:00.934274] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 05:00:00.934739] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 05:00:00.935266] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 05:10:01.437319] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 05:10:01.437783] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 05:10:01.438361] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 05:20:01.532562] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 05:20:01.536813] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 05:20:01.537338] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 05:30:02.335091] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 05:30:02.335439] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 05:30:02.336074] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 05:40:02.890002] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 05:40:02.893148] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 05:40:02.893581] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 05:50:02.950104] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 05:50:02.950769] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 05:50:02.953615] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 06:00:03.588240] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 06:00:03.590598] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 06:00:03.591183] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 06:10:04.368233] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 06:10:04.368541] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 06:10:04.368861] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 06:20:05.214175] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 06:20:05.217829] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 06:20:05.218233] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 06:30:06.129286] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 06:30:06.135310] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 06:30:06.135854] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 06:40:06.425366] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 06:40:06.425559] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 06:40:06.426379] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 06:50:06.776753] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 06:50:06.781422] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 06:50:06.781873] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 07:00:07.599484] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 07:00:07.607985] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 07:00:07.608483] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 07:10:08.459851] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 07:10:08.461308] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 07:10:08.461907] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 07:20:09.026367] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 07:20:09.030657] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 07:20:09.031033] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 07:30:09.035309] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 07:30:10.037934] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 07:30:10.040472] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 07:40:09.608936] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 07:40:10.620794] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 07:40:10.622823] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 07:50:10.477401] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 07:50:11.490292] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 07:50:11.499353] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 08:00:11.370131] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 08:00:12.380371] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 08:00:12.381105] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 08:10:12.056821] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 08:10:13.062728] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 08:10:13.063247] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 08:20:12.493576] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 08:20:13.513890] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 08:20:13.515852] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 08:30:13.503193] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 08:30:14.516230] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 08:30:14.522069] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 08:40:13.837517] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 08:40:14.847546] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 08:40:14.848237] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 08:50:13.981015] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 08:50:14.991292] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 08:50:14.991983] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 09:00:14.852298] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 09:00:15.870093] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 09:00:15.882465] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 09:10:15.779219] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 09:10:16.796278] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 09:10:16.797867] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 09:20:16.507024] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 09:20:17.517471] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 09:20:17.522863] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 09:30:17.131983] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 09:30:18.137648] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 09:30:18.138384] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 09:40:17.677965] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 09:40:18.688179] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 09:40:18.688666] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 09:50:18.401355] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 09:50:19.407105] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 09:50:19.407640] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 10:00:19.015897] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 10:00:20.029086] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 10:00:20.031686] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 10:10:19.550739] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 10:10:20.560745] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 10:10:20.561190] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 10:20:20.307484] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 10:20:21.322038] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 10:20:21.323351] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 10:30:20.871294] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 10:30:21.881902] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 10:30:21.882835] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 10:40:21.660309] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 10:40:22.669903] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 10:40:22.672855] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 10:50:22.178866] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 10:50:23.188289] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 10:50:23.188936] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 11:00:22.937699] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 11:00:23.946213] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 11:00:23.950165] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 11:10:23.641808] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 11:10:24.652034] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 11:10:24.653125] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 11:20:24.172091] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 11:20:25.189420] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 11:20:25.191615] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 11:30:24.191995] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 11:30:25.207715] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 11:30:25.215741] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 11:40:24.778271] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 11:40:25.783433] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 11:40:25.785825] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 11:50:25.226481] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 11:50:26.236635] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 11:50:26.242846] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 12:00:25.334063] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 12:00:26.339921] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 12:00:26.340406] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 12:10:25.834607] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 12:10:26.844796] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 12:10:26.845534] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 12:20:26.683744] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 12:20:27.693911] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 12:20:27.694628] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 12:30:27.284730] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 12:30:28.299586] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 12:30:28.302213] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 12:40:27.735277] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 12:40:28.750470] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 12:40:28.755667] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 12:50:28.679107] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 12:50:29.685226] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 12:50:29.686937] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 13:00:29.055909] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 13:00:30.065288] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 13:00:30.065557] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 13:10:29.760644] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 13:10:30.770821] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 13:10:30.771649] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 13:20:30.204347] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 13:20:31.214515] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 13:20:31.215049] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 13:30:30.793486] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 13:30:31.799073] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 13:30:31.799542] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 13:40:31.518073] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 13:40:32.531017] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 13:40:32.533888] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 13:50:32.289339] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 13:50:33.298604] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 13:50:33.300102] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 14:00:32.671950] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 14:00:33.678006] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 14:00:33.678818] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 14:10:32.705785] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 14:10:33.711376] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 14:10:33.711770] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 14:20:32.911601] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 14:20:33.921629] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 14:20:33.927177] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 14:30:33.700297] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 14:30:34.717019] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 14:30:34.725495] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 14:40:34.668401] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 14:40:35.682157] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 14:40:35.684080] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 14:50:34.713208] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 14:50:35.723489] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 14:50:35.724146] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 15:00:35.160224] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 15:00:36.170470] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 15:00:36.171130] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 15:10:35.734953] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 15:10:36.751113] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 15:10:36.758442] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 15:20:36.282529] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 15:20:37.292622] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 15:20:37.293277] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 15:30:36.760069] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 15:30:37.773759] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 15:30:37.779183] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 15:40:37.409805] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 15:40:38.426739] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 15:40:38.434183] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 15:50:37.894095] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 15:50:38.904001] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 15:50:38.904359] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 16:00:38.459768] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 16:00:39.479352] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 16:00:39.479739] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 16:10:38.841429] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 16:10:39.851467] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 16:10:39.852260] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 16:20:39.319027] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 16:20:40.328853] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 16:20:40.332157] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 16:30:39.704489] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 16:30:40.714641] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 16:30:40.715235] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 16:40:40.123431] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 16:40:41.141531] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 16:40:41.147706] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 16:50:40.643217] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 16:50:41.653559] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 16:50:41.662860] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 17:00:41.152064] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 17:00:42.157487] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 17:00:42.157891] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 17:10:41.564401] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 17:10:42.574629] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 17:10:42.580133] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 17:20:42.064504] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 17:20:43.074400] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 17:20:43.074959] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 17:30:42.531424] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 17:30:43.543071] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 17:30:43.545025] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 17:40:42.981204] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 17:40:43.999848] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 17:40:44.007736] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 17:50:43.564494] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 17:50:44.574570] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 17:50:44.574928] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 18:00:43.888663] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 18:00:44.903924] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 18:00:44.904339] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 18:10:44.396257] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 18:10:45.406172] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 18:10:45.406735] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 18:20:44.887086] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 18:20:45.897059] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 18:20:45.897587] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 18:30:45.428075] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 18:30:46.438468] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 18:30:46.444081] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 18:40:45.862306] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 18:40:46.878537] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 18:40:46.885711] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 18:50:46.296577] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 18:50:47.306518] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 18:50:47.306953] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 19:00:46.724675] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 19:00:47.744268] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 19:00:47.744853] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 19:10:47.116018] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 19:10:48.125929] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 19:10:48.126172] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 19:20:47.683874] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 19:20:48.693996] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 19:20:48.699582] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 19:30:48.222170] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 19:30:49.238348] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 19:30:49.244139] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 19:40:48.657604] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 19:40:49.677821] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 19:40:49.678447] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 19:50:49.124405] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 19:50:50.134322] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 19:50:50.135958] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 20:00:49.630317] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 20:00:50.639366] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 20:00:50.639832] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 20:10:50.006077] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 20:10:51.016484] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 20:10:51.016818] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 20:20:50.463088] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 20:20:51.473076] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 20:20:51.473668] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 20:30:51.016623] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 20:30:52.026726] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 20:30:52.032186] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 20:40:51.479298] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 20:40:52.489966] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 20:40:52.490446] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 20:50:52.043560] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 20:50:53.053966] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 20:50:53.054729] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 21:00:52.478524] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 21:00:53.496308] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 21:00:53.500431] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 21:10:19.726160] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2012-06-05 21:10:53.026093] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 21:10:54.036004] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 21:10:54.036434] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 21:20:53.602246] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 21:20:54.615420] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 21:20:54.617947] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 21:30:54.077244] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 21:30:55.085919] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 21:30:55.091006] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 21:40:54.546686] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 21:40:55.552464] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 21:40:55.553022] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 21:50:55.114735] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 21:50:56.124779] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 21:50:56.130301] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 22:00:55.694484] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 22:00:56.700196] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 22:00:56.700726] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 22:10:56.180551] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 22:10:57.198578] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 22:10:57.204404] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 22:20:56.737088] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 22:20:57.753094] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 22:20:57.757543] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 22:30:57.186287] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 22:30:58.196418] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 22:30:58.197010] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 22:40:57.661098] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 22:40:58.673894] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 22:40:58.680561] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 22:50:58.186509] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 22:50:59.196742] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 22:50:59.197188] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 23:00:58.769710] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 23:00:59.785623] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 23:00:59.791792] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 23:10:59.275857] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 23:11:00.286062] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 23:11:00.286838] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 23:20:59.852117] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 23:21:00.862451] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 23:21:00.862727] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 23:31:00.343437] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 23:31:01.357422] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 23:31:01.357899] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 23:41:00.877107] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 23:41:01.887471] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 23:41:01.887839] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-05 23:51:01.411227] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-05 23:51:02.421749] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-05 23:51:02.427431] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 00:01:01.956937] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 00:01:02.974690] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 00:01:02.977251] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 00:11:02.395533] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 00:11:03.407321] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 00:11:03.409388] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 00:21:03.007928] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 00:21:04.018010] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 00:21:04.027551] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 00:31:03.534833] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 00:31:04.545000] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 00:31:04.545630] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 00:41:04.096424] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 00:41:05.106468] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 00:41:05.111903] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 00:51:04.539277] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 00:51:05.549339] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 00:51:05.550001] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 01:01:05.100219] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 01:01:06.109697] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 01:01:06.113112] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 01:11:05.594815] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 01:11:06.612749] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 01:11:06.621994] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 01:21:06.093517] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 01:21:07.103651] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 01:21:07.104164] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 01:31:06.695001] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 01:31:07.705379] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 01:31:07.710824] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 01:41:07.232581] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 01:41:08.248433] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 01:41:08.248995] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 01:51:07.701672] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 01:51:08.711802] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 01:51:08.712128] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 02:01:08.210614] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down
>[2012-06-06 02:01:09.220636] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down
>[2012-06-06 02:01:09.226512] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down
>[2012-06-06 02:05:31.465544] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3271ee570d] (-->/lib64/libpthread.so.0() [0x32722077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407c18]))) 0-: received signum (15), shutting down
>[2012-06-06 02:05:32.489444] I [glusterfsd.c:1666:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.3.0qa45
>[2012-06-06 02:05:32.513859] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-2: adding option 'node-uuid' for volume 'vol-replicate-2' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d'
>[2012-06-06 02:05:32.513921] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-1: adding option 'node-uuid' for volume 'vol-replicate-1' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d'
>[2012-06-06 02:05:32.513959] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-0: adding option 'node-uuid' for volume 'vol-replicate-0' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d'
>[2012-06-06 02:05:32.592965] I [client.c:2142:notify] 0-vol-client-0: parent translators are ready, attempting connect on transport
>[2012-06-06 02:05:32.598093] I [client.c:2142:notify] 0-vol-client-1: parent translators are ready, attempting connect on transport
>[2012-06-06 02:05:32.603362] I [client.c:2142:notify] 0-vol-client-2: parent translators are ready, attempting connect on transport
>[2012-06-06 02:05:32.608304] I [client.c:2142:notify] 0-vol-client-3: parent translators are ready, attempting connect on transport
>[2012-06-06 02:05:32.613078] I [client.c:2142:notify] 0-vol-client-4: parent translators are ready, attempting connect on transport
>[2012-06-06 02:05:32.618376] I [client.c:2142:notify] 0-vol-client-5: parent translators are ready, attempting connect on transport
>[2012-06-06 02:05:32.623345] I [client.c:2142:notify] 0-vol-client-6: parent translators are ready, attempting connect on transport
>[2012-06-06 02:05:32.628233] I [client.c:2142:notify] 0-vol-client-7: parent translators are ready, attempting connect on transport
>[2012-06-06 02:05:32.633249] I [client.c:2142:notify] 0-vol-client-8: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume vol-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.16.159.184
>  4:     option remote-subvolume /export_b1/dir1
>  5:     option transport-type tcp
>  6:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
>  7:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
>  8: end-volume
>  9: 
> 10: volume vol-client-1
> 11:     type protocol/client
> 12:     option remote-host 10.16.159.188
> 13:     option remote-subvolume /export_b1/dir1
> 14:     option transport-type tcp
> 15:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 16:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 17: end-volume
> 18: 
> 19: volume vol-client-2
> 20:     type protocol/client
> 21:     option remote-host 10.16.159.196
> 22:     option remote-subvolume /export_b1/dir1
> 23:     option transport-type tcp
> 24:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 25:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 26: end-volume
> 27: 
> 28: volume vol-client-3
> 29:     type protocol/client
> 30:     option remote-host 10.16.159.184
> 31:     option remote-subvolume /export_c1/dir1
> 32:     option transport-type tcp
> 33:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 34:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 35: end-volume
> 36: 
> 37: volume vol-client-4
> 38:     type protocol/client
> 39:     option remote-host 10.16.159.188
> 40:     option remote-subvolume /export_c1/dir1
> 41:     option transport-type tcp
> 42:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 43:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 44: end-volume
> 45: 
> 46: volume vol-client-5
> 47:     type protocol/client
> 48:     option remote-host 10.16.159.196
> 49:     option remote-subvolume /export_c1/dir1
> 50:     option transport-type tcp
> 51:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 52:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 53: end-volume
> 54: 
> 55: volume vol-client-6
> 56:     type protocol/client
> 57:     option remote-host 10.16.159.184
> 58:     option remote-subvolume /export_d1/dir1
> 59:     option transport-type tcp
> 60:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 61:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 62: end-volume
> 63: 
> 64: volume vol-client-7
> 65:     type protocol/client
> 66:     option remote-host 10.16.159.188
> 67:     option remote-subvolume /export_d1/dir1
> 68:     option transport-type tcp
> 69:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 70:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 71: end-volume
> 72: 
> 73: volume vol-client-8
> 74:     type protocol/client
> 75:     option remote-host 10.16.159.196
> 76:     option remote-subvolume /export_d1/dir1
> 77:     option transport-type tcp
> 78:     option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52
> 79:     option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e
> 80: end-volume
> 81: 
> 82: volume vol-replicate-0
> 83:     type cluster/replicate
> 84:     option background-self-heal-count 0
> 85:     option metadata-self-heal on
> 86:     option data-self-heal on
> 87:     option entry-self-heal on
> 88:     option self-heal-daemon on
> 89:     option iam-self-heal-daemon yes
> 90:     subvolumes vol-client-0 vol-client-1 vol-client-2
> 91: end-volume
> 92: 
> 93: volume vol-replicate-1
> 94:     type cluster/replicate
> 95:     option background-self-heal-count 0
> 96:     option metadata-self-heal on
> 97:     option data-self-heal on
> 98:     option entry-self-heal on
> 99:     option self-heal-daemon on
>100:     option iam-self-heal-daemon yes
>101:     subvolumes vol-client-3 vol-client-4 vol-client-5
>102: end-volume
>103: 
>104: volume vol-replicate-2
>105:     type cluster/replicate
>106:     option background-self-heal-count 0
>107:     option metadata-self-heal on
>108:     option data-self-heal on
>109:     option entry-self-heal on
>110:     option self-heal-daemon on
>111:     option iam-self-heal-daemon yes
>112:     subvolumes vol-client-6 vol-client-7 vol-client-8
>113: end-volume
>114: 
>115: volume glustershd
>116:     type debug/io-stats
>117:     subvolumes vol-replicate-0 vol-replicate-1 vol-replicate-2
>118: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-06 02:05:32.640966] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-0: changing port to 24009 (from 0)
>[2012-06-06 02:05:32.641149] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-1: changing port to 24009 (from 0)
>[2012-06-06 02:05:32.641308] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-2: changing port to 24009 (from 0)
>[2012-06-06 02:05:32.641395] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-3: changing port to 24010 (from 0)
>[2012-06-06 02:05:32.641473] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-4: changing port to 24010 (from 0)
>[2012-06-06 02:05:32.641649] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-5: changing port to 24010 (from 0)
>[2012-06-06 02:05:32.641734] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-6: changing port to 24011 (from 0)
>[2012-06-06 02:05:32.641822] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-7: changing port to 24011 (from 0)
>[2012-06-06 02:05:32.642407] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-8: changing port to 24011 (from 0)
>[2012-06-06 02:05:36.174004] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-06-06 02:05:36.174103] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-06-06 02:05:36.516990] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.517666] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-0: Connected to 10.16.159.184:24009, attached to remote volume '/export_b1/dir1'.
>[2012-06-06 02:05:36.517714] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.517819] I [afr-common.c:3627:afr_notify] 0-vol-replicate-0: Subvolume 'vol-client-0' came back up; going online.
>[2012-06-06 02:05:36.518253] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-0: Server lk version = 1
>[2012-06-06 02:05:36.521887] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.522243] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-1: Connected to 10.16.159.188:24009, attached to remote volume '/export_b1/dir1'.
>[2012-06-06 02:05:36.522275] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.522861] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-1: Server lk version = 1
>[2012-06-06 02:05:36.527842] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.528453] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-2: Connected to 10.16.159.196:24009, attached to remote volume '/export_b1/dir1'.
>[2012-06-06 02:05:36.528537] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.535155] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-2: Server lk version = 1
>[2012-06-06 02:05:36.536274] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.537446] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-3: Connected to 10.16.159.184:24010, attached to remote volume '/export_c1/dir1'.
>[2012-06-06 02:05:36.537485] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.537767] I [afr-common.c:3627:afr_notify] 0-vol-replicate-1: Subvolume 'vol-client-3' came back up; going online.
>[2012-06-06 02:05:36.539003] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-3: Server lk version = 1
>[2012-06-06 02:05:36.539732] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-4: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.540463] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-4: Connected to 10.16.159.188:24010, attached to remote volume '/export_c1/dir1'.
>[2012-06-06 02:05:36.540499] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.541835] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-4: Server lk version = 1
>[2012-06-06 02:05:36.547427] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-5: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.548204] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-5: Connected to 10.16.159.196:24010, attached to remote volume '/export_c1/dir1'.
>[2012-06-06 02:05:36.548243] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.552824] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-5: Server lk version = 1
>[2012-06-06 02:05:36.553418] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-6: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.554048] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-6: Connected to 10.16.159.184:24011, attached to remote volume '/export_d1/dir1'.
>[2012-06-06 02:05:36.554079] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.554201] I [afr-common.c:3627:afr_notify] 0-vol-replicate-2: Subvolume 'vol-client-6' came back up; going online.
>[2012-06-06 02:05:36.555330] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-6: Server lk version = 1
>[2012-06-06 02:05:36.557352] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-7: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.557697] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-7: Connected to 10.16.159.188:24011, attached to remote volume '/export_d1/dir1'.
>[2012-06-06 02:05:36.557730] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.558864] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-7: Server lk version = 1
>[2012-06-06 02:05:36.563341] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-8: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-06 02:05:36.563931] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-8: Connected to 10.16.159.196:24011, attached to remote volume '/export_d1/dir1'.
>[2012-06-06 02:05:36.564005] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-06 02:05:36.564945] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-8: Server lk version = 1
>pending frames:
>
>patchset: git://git.gluster.com/glusterfs.git
>signal received: 11
>time of crash: 2012-06-06 02:12:08
>configuration details:
>argp 1
>backtrace 1
>dlfcn 1
>fdatasync 1
>libpthread 1
>llistxattr 1
>setfsid 1
>spinlock 1
>epoll.h 1
>xattr.h 1
>st_atim.tv_nsec 1
>package-string: glusterfs 3.3.0qa45
>/lib64/libc.so.6[0x3271e32900]
>/lib64/libc.so.6[0x3271f14f99]
>/lib64/libc.so.6(xdr_u_int+0x53)[0x3271f141f3]
>/lib64/libc.so.6(xdr_string+0x48)[0x3271f148e8]
>/usr/local/lib/libgfxdr.so.0(xdr_gd1_mgmt_brick_op_req+0x29)[0x7f3af77c5c67]
>/usr/local/lib/libgfxdr.so.0(xdr_to_generic+0x81)[0x7f3af77bd71f]
>/usr/local/sbin/glusterfs(glusterfs_handle_translator_op+0x13e)[0x40a4b7]
>/usr/local/lib/libglusterfs.so.0(synctask_wrap+0x38)[0x7f3af7c4a882]
>/lib64/libc.so.6[0x3271e43610]
>---------

Actions: View

Attachments on bug 829168: 589744