Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 589744 Details for
Bug 829168
self-heal daemon process crashed during self-heal of 50k files
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
glustershd log
glustershd.log (text/x-log), 85.11 KB, created by
Shwetha Panduranga
on 2012-06-06 06:41:37 UTC
(
hide
)
Description:
glustershd log
Filename:
MIME Type:
Creator:
Shwetha Panduranga
Created:
2012-06-06 06:41:37 UTC
Size:
85.11 KB
patch
obsolete
>[2012-06-05 03:49:49.178292] I [glusterfsd.c:1666:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.3.0qa45 >[2012-06-05 03:49:49.229770] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-2: adding option 'node-uuid' for volume 'vol-replicate-2' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d' >[2012-06-05 03:49:49.229811] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-1: adding option 'node-uuid' for volume 'vol-replicate-1' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d' >[2012-06-05 03:49:49.229823] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-0: adding option 'node-uuid' for volume 'vol-replicate-0' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d' >[2012-06-05 03:49:49.249452] I [client.c:2142:notify] 0-vol-client-0: parent translators are ready, attempting connect on transport >[2012-06-05 03:49:49.252189] I [client.c:2142:notify] 0-vol-client-1: parent translators are ready, attempting connect on transport >[2012-06-05 03:49:49.255765] I [client.c:2142:notify] 0-vol-client-2: parent translators are ready, attempting connect on transport >[2012-06-05 03:49:49.258421] I [client.c:2142:notify] 0-vol-client-3: parent translators are ready, attempting connect on transport >[2012-06-05 03:49:49.261057] I [client.c:2142:notify] 0-vol-client-4: parent translators are ready, attempting connect on transport >[2012-06-05 03:49:49.264776] I [client.c:2142:notify] 0-vol-client-5: parent translators are ready, attempting connect on transport >[2012-06-05 03:49:49.268040] I [client.c:2142:notify] 0-vol-client-6: parent translators are ready, attempting connect on transport >[2012-06-05 03:49:49.274895] I [client.c:2142:notify] 0-vol-client-7: parent translators are ready, attempting connect on transport >[2012-06-05 03:49:49.281484] I [client.c:2142:notify] 0-vol-client-8: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume vol-client-0 > 2: type protocol/client > 3: option remote-host 10.16.159.184 > 4: option remote-subvolume /export_b1/dir1 > 5: option transport-type tcp > 6: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 7: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 8: end-volume > 9: > 10: volume vol-client-1 > 11: type protocol/client > 12: option remote-host 10.16.159.188 > 13: option remote-subvolume /export_b1/dir1 > 14: option transport-type tcp > 15: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 16: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 17: end-volume > 18: > 19: volume vol-client-2 > 20: type protocol/client > 21: option remote-host 10.16.159.196 > 22: option remote-subvolume /export_b1/dir1 > 23: option transport-type tcp > 24: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 25: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 26: end-volume > 27: > 28: volume vol-client-3 > 29: type protocol/client > 30: option remote-host 10.16.159.184 > 31: option remote-subvolume /export_c1/dir1 > 32: option transport-type tcp > 33: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 34: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 35: end-volume > 36: > 37: volume vol-client-4 > 38: type protocol/client > 39: option remote-host 10.16.159.188 > 40: option remote-subvolume /export_c1/dir1 > 41: option transport-type tcp > 42: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 43: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 44: end-volume > 45: > 46: volume vol-client-5 > 47: type protocol/client > 48: option remote-host 10.16.159.196 > 49: option remote-subvolume /export_c1/dir1 > 50: option transport-type tcp > 51: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 52: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 53: end-volume > 54: > 55: volume vol-client-6 > 56: type protocol/client > 57: option remote-host 10.16.159.184 > 58: option remote-subvolume /export_d1/dir1 > 59: option transport-type tcp > 60: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 61: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 62: end-volume > 63: > 64: volume vol-client-7 > 65: type protocol/client > 66: option remote-host 10.16.159.188 > 67: option remote-subvolume /export_d1/dir1 > 68: option transport-type tcp > 69: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 70: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 71: end-volume > 72: > 73: volume vol-client-8 > 74: type protocol/client > 75: option remote-host 10.16.159.196 > 76: option remote-subvolume /export_d1/dir1 > 77: option transport-type tcp > 78: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 79: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 80: end-volume > 81: > 82: volume vol-replicate-0 > 83: type cluster/replicate > 84: option background-self-heal-count 0 > 85: option metadata-self-heal on > 86: option data-self-heal on > 87: option entry-self-heal on > 88: option self-heal-daemon on > 89: option iam-self-heal-daemon yes > 90: subvolumes vol-client-0 vol-client-1 vol-client-2 > 91: end-volume > 92: > 93: volume vol-replicate-1 > 94: type cluster/replicate > 95: option background-self-heal-count 0 > 96: option metadata-self-heal on > 97: option data-self-heal on > 98: option entry-self-heal on > 99: option self-heal-daemon on >100: option iam-self-heal-daemon yes >101: subvolumes vol-client-3 vol-client-4 vol-client-5 >102: end-volume >103: >104: volume vol-replicate-2 >105: type cluster/replicate >106: option background-self-heal-count 0 >107: option metadata-self-heal on >108: option data-self-heal on >109: option entry-self-heal on >110: option self-heal-daemon on >111: option iam-self-heal-daemon yes >112: subvolumes vol-client-6 vol-client-7 vol-client-8 >113: end-volume >114: >115: volume glustershd >116: type debug/io-stats >117: subvolumes vol-replicate-0 vol-replicate-1 vol-replicate-2 >118: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-05 03:49:49.287840] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-0: changing port to 24009 (from 0) >[2012-06-05 03:49:49.287981] E [client-handshake.c:1717:client_query_portmap_cbk] 0-vol-client-1: failed to get the port number for remote subvolume >[2012-06-05 03:49:49.288091] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-3: changing port to 24010 (from 0) >[2012-06-05 03:49:49.288217] E [client-handshake.c:1717:client_query_portmap_cbk] 0-vol-client-4: failed to get the port number for remote subvolume >[2012-06-05 03:49:49.288893] I [client.c:2090:client_rpc_notify] 0-vol-client-1: disconnected >[2012-06-05 03:49:49.289038] I [client.c:2090:client_rpc_notify] 0-vol-client-4: disconnected >[2012-06-05 03:49:49.289125] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-6: changing port to 24011 (from 0) >[2012-06-05 03:49:49.289251] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-2: changing port to 24009 (from 0) >[2012-06-05 03:49:49.289412] E [client-handshake.c:1717:client_query_portmap_cbk] 0-vol-client-7: failed to get the port number for remote subvolume >[2012-06-05 03:49:49.289740] I [client.c:2090:client_rpc_notify] 0-vol-client-7: disconnected >[2012-06-05 03:49:49.289832] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-5: changing port to 24010 (from 0) >[2012-06-05 03:49:49.289991] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-8: changing port to 24011 (from 0) >[2012-06-05 03:49:53.021102] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-06-05 03:49:53.021180] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-06-05 03:49:53.192702] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:53.193499] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-0: Connected to 10.16.159.184:24009, attached to remote volume '/export_b1/dir1'. >[2012-06-05 03:49:53.193562] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:53.193705] I [afr-common.c:3627:afr_notify] 0-vol-replicate-0: Subvolume 'vol-client-0' came back up; going online. >[2012-06-05 03:49:53.194080] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-0: Server lk version = 1 >[2012-06-05 03:49:53.201215] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-1: changing port to 24009 (from 0) >[2012-06-05 03:49:53.210293] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:53.211016] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-3: Connected to 10.16.159.184:24010, attached to remote volume '/export_c1/dir1'. >[2012-06-05 03:49:53.211074] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:53.211219] I [afr-common.c:3627:afr_notify] 0-vol-replicate-1: Subvolume 'vol-client-3' came back up; going online. >[2012-06-05 03:49:53.211583] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-3: Server lk version = 1 >[2012-06-05 03:49:53.217711] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-4: changing port to 24010 (from 0) >[2012-06-05 03:49:53.225556] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-6: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:53.226209] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-6: Connected to 10.16.159.184:24011, attached to remote volume '/export_d1/dir1'. >[2012-06-05 03:49:53.226257] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:53.226404] I [afr-common.c:3627:afr_notify] 0-vol-replicate-2: Subvolume 'vol-client-6' came back up; going online. >[2012-06-05 03:49:53.229414] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-6: Server lk version = 1 >[2012-06-05 03:49:53.233557] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:53.234423] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-2: Connected to 10.16.159.196:24009, attached to remote volume '/export_b1/dir1'. >[2012-06-05 03:49:53.234473] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:53.235378] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-2: Server lk version = 1 >[2012-06-05 03:49:53.240898] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-7: changing port to 24011 (from 0) >[2012-06-05 03:49:53.248910] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-5: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:53.249626] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-5: Connected to 10.16.159.196:24010, attached to remote volume '/export_c1/dir1'. >[2012-06-05 03:49:53.249861] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:53.251687] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-5: Server lk version = 1 >[2012-06-05 03:49:53.257387] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-8: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:53.258008] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-8: Connected to 10.16.159.196:24011, attached to remote volume '/export_d1/dir1'. >[2012-06-05 03:49:53.258044] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:53.260705] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-8: Server lk version = 1 >[2012-06-05 03:49:56.262358] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:56.262875] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-1: Connected to 10.16.159.188:24009, attached to remote volume '/export_b1/dir1'. >[2012-06-05 03:49:56.262911] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:56.263573] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-1: Server lk version = 1 >[2012-06-05 03:49:56.267918] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-4: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:56.268425] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-4: Connected to 10.16.159.188:24010, attached to remote volume '/export_c1/dir1'. >[2012-06-05 03:49:56.268459] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:56.269091] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-4: Server lk version = 1 >[2012-06-05 03:49:56.274539] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-7: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-05 03:49:56.274998] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-7: Connected to 10.16.159.188:24011, attached to remote volume '/export_d1/dir1'. >[2012-06-05 03:49:56.275033] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-05 03:49:56.275730] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-7: Server lk version = 1 >[2012-06-05 03:50:04.679071] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.188:24009) >[2012-06-05 03:50:04.679190] I [client.c:2090:client_rpc_notify] 0-vol-client-1: disconnected >[2012-06-05 03:50:12.908907] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-4: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.188:24010) >[2012-06-05 03:50:12.909038] I [client.c:2090:client_rpc_notify] 0-vol-client-4: disconnected >[2012-06-05 03:50:15.283671] E [socket.c:1715:socket_connect_finish] 0-vol-client-1: connection to 10.16.159.188:24009 failed (Connection refused) >[2012-06-05 03:50:21.717155] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-7: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.188:24011) >[2012-06-05 03:50:21.717263] I [client.c:2090:client_rpc_notify] 0-vol-client-7: disconnected >[2012-06-05 03:50:23.299485] E [socket.c:1715:socket_connect_finish] 0-vol-client-4: connection to 10.16.159.188:24010 failed (Connection refused) >[2012-06-05 03:50:32.320252] E [socket.c:1715:socket_connect_finish] 0-vol-client-7: connection to 10.16.159.188:24011 failed (Connection refused) >[2012-06-05 03:50:41.749353] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.196:24009) >[2012-06-05 03:50:41.749503] I [client.c:2090:client_rpc_notify] 0-vol-client-2: disconnected >[2012-06-05 03:50:47.653867] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-5: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.196:24010) >[2012-06-05 03:50:47.653979] I [client.c:2090:client_rpc_notify] 0-vol-client-5: disconnected >[2012-06-05 03:50:52.421570] E [socket.c:1715:socket_connect_finish] 0-vol-client-2: connection to 10.16.159.196:24009 failed (Connection refused) >[2012-06-05 03:50:52.675259] W [socket.c:1512:__socket_proto_state_machine] 0-vol-client-8: reading from socket failed. Error (Transport endpoint is not connected), peer (10.16.159.196:24011) >[2012-06-05 03:50:52.675451] I [client.c:2090:client_rpc_notify] 0-vol-client-8: disconnected >[2012-06-05 03:50:58.460806] E [socket.c:1715:socket_connect_finish] 0-vol-client-5: connection to 10.16.159.196:24010 failed (Connection refused) >[2012-06-05 03:51:03.501953] E [socket.c:1715:socket_connect_finish] 0-vol-client-8: connection to 10.16.159.196:24011 failed (Connection refused) >[2012-06-05 03:59:57.036775] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 03:59:57.039657] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 03:59:57.039675] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 04:09:57.729266] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 04:09:57.729405] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 04:09:57.729869] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 04:19:58.244458] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 04:19:58.244643] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 04:19:58.245888] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 04:29:58.919633] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 04:29:58.920034] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 04:29:58.925338] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 04:39:59.467531] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 04:39:59.470316] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 04:39:59.470687] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 04:50:00.088726] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 04:50:00.091523] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 04:50:00.091904] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 05:00:00.934274] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 05:00:00.934739] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 05:00:00.935266] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 05:10:01.437319] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 05:10:01.437783] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 05:10:01.438361] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 05:20:01.532562] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 05:20:01.536813] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 05:20:01.537338] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 05:30:02.335091] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 05:30:02.335439] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 05:30:02.336074] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 05:40:02.890002] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 05:40:02.893148] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 05:40:02.893581] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 05:50:02.950104] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 05:50:02.950769] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 05:50:02.953615] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 06:00:03.588240] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 06:00:03.590598] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 06:00:03.591183] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 06:10:04.368233] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 06:10:04.368541] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 06:10:04.368861] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 06:20:05.214175] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 06:20:05.217829] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 06:20:05.218233] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 06:30:06.129286] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 06:30:06.135310] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 06:30:06.135854] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 06:40:06.425366] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 06:40:06.425559] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 06:40:06.426379] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 06:50:06.776753] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 06:50:06.781422] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 06:50:06.781873] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 07:00:07.599484] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 07:00:07.607985] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 07:00:07.608483] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 07:10:08.459851] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 07:10:08.461308] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 07:10:08.461907] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 07:20:09.026367] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 07:20:09.030657] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 07:20:09.031033] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 07:30:09.035309] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 07:30:10.037934] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 07:30:10.040472] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 07:40:09.608936] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 07:40:10.620794] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 07:40:10.622823] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 07:50:10.477401] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 07:50:11.490292] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 07:50:11.499353] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 08:00:11.370131] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 08:00:12.380371] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 08:00:12.381105] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 08:10:12.056821] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 08:10:13.062728] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 08:10:13.063247] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 08:20:12.493576] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 08:20:13.513890] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 08:20:13.515852] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 08:30:13.503193] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 08:30:14.516230] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 08:30:14.522069] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 08:40:13.837517] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 08:40:14.847546] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 08:40:14.848237] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 08:50:13.981015] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 08:50:14.991292] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 08:50:14.991983] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 09:00:14.852298] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 09:00:15.870093] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 09:00:15.882465] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 09:10:15.779219] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 09:10:16.796278] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 09:10:16.797867] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 09:20:16.507024] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 09:20:17.517471] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 09:20:17.522863] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 09:30:17.131983] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 09:30:18.137648] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 09:30:18.138384] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 09:40:17.677965] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 09:40:18.688179] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 09:40:18.688666] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 09:50:18.401355] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 09:50:19.407105] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 09:50:19.407640] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 10:00:19.015897] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 10:00:20.029086] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 10:00:20.031686] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 10:10:19.550739] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 10:10:20.560745] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 10:10:20.561190] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 10:20:20.307484] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 10:20:21.322038] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 10:20:21.323351] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 10:30:20.871294] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 10:30:21.881902] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 10:30:21.882835] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 10:40:21.660309] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 10:40:22.669903] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 10:40:22.672855] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 10:50:22.178866] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 10:50:23.188289] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 10:50:23.188936] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 11:00:22.937699] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 11:00:23.946213] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 11:00:23.950165] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 11:10:23.641808] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 11:10:24.652034] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 11:10:24.653125] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 11:20:24.172091] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 11:20:25.189420] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 11:20:25.191615] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 11:30:24.191995] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 11:30:25.207715] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 11:30:25.215741] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 11:40:24.778271] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 11:40:25.783433] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 11:40:25.785825] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 11:50:25.226481] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 11:50:26.236635] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 11:50:26.242846] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 12:00:25.334063] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 12:00:26.339921] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 12:00:26.340406] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 12:10:25.834607] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 12:10:26.844796] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 12:10:26.845534] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 12:20:26.683744] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 12:20:27.693911] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 12:20:27.694628] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 12:30:27.284730] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 12:30:28.299586] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 12:30:28.302213] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 12:40:27.735277] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 12:40:28.750470] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 12:40:28.755667] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 12:50:28.679107] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 12:50:29.685226] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 12:50:29.686937] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 13:00:29.055909] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 13:00:30.065288] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 13:00:30.065557] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 13:10:29.760644] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 13:10:30.770821] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 13:10:30.771649] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 13:20:30.204347] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 13:20:31.214515] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 13:20:31.215049] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 13:30:30.793486] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 13:30:31.799073] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 13:30:31.799542] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 13:40:31.518073] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 13:40:32.531017] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 13:40:32.533888] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 13:50:32.289339] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 13:50:33.298604] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 13:50:33.300102] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 14:00:32.671950] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 14:00:33.678006] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 14:00:33.678818] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 14:10:32.705785] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 14:10:33.711376] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 14:10:33.711770] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 14:20:32.911601] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 14:20:33.921629] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 14:20:33.927177] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 14:30:33.700297] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 14:30:34.717019] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 14:30:34.725495] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 14:40:34.668401] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 14:40:35.682157] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 14:40:35.684080] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 14:50:34.713208] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 14:50:35.723489] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 14:50:35.724146] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 15:00:35.160224] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 15:00:36.170470] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 15:00:36.171130] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 15:10:35.734953] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 15:10:36.751113] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 15:10:36.758442] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 15:20:36.282529] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 15:20:37.292622] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 15:20:37.293277] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 15:30:36.760069] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 15:30:37.773759] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 15:30:37.779183] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 15:40:37.409805] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 15:40:38.426739] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 15:40:38.434183] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 15:50:37.894095] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 15:50:38.904001] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 15:50:38.904359] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 16:00:38.459768] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 16:00:39.479352] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 16:00:39.479739] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 16:10:38.841429] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 16:10:39.851467] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 16:10:39.852260] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 16:20:39.319027] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 16:20:40.328853] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 16:20:40.332157] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 16:30:39.704489] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 16:30:40.714641] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 16:30:40.715235] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 16:40:40.123431] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 16:40:41.141531] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 16:40:41.147706] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 16:50:40.643217] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 16:50:41.653559] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 16:50:41.662860] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 17:00:41.152064] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 17:00:42.157487] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 17:00:42.157891] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 17:10:41.564401] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 17:10:42.574629] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 17:10:42.580133] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 17:20:42.064504] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 17:20:43.074400] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 17:20:43.074959] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 17:30:42.531424] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 17:30:43.543071] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 17:30:43.545025] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 17:40:42.981204] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 17:40:43.999848] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 17:40:44.007736] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 17:50:43.564494] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 17:50:44.574570] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 17:50:44.574928] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 18:00:43.888663] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 18:00:44.903924] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 18:00:44.904339] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 18:10:44.396257] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 18:10:45.406172] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 18:10:45.406735] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 18:20:44.887086] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 18:20:45.897059] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 18:20:45.897587] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 18:30:45.428075] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 18:30:46.438468] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 18:30:46.444081] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 18:40:45.862306] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 18:40:46.878537] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 18:40:46.885711] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 18:50:46.296577] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 18:50:47.306518] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 18:50:47.306953] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 19:00:46.724675] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 19:00:47.744268] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 19:00:47.744853] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 19:10:47.116018] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 19:10:48.125929] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 19:10:48.126172] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 19:20:47.683874] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 19:20:48.693996] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 19:20:48.699582] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 19:30:48.222170] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 19:30:49.238348] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 19:30:49.244139] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 19:40:48.657604] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 19:40:49.677821] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 19:40:49.678447] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 19:50:49.124405] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 19:50:50.134322] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 19:50:50.135958] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 20:00:49.630317] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 20:00:50.639366] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 20:00:50.639832] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 20:10:50.006077] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 20:10:51.016484] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 20:10:51.016818] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 20:20:50.463088] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 20:20:51.473076] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 20:20:51.473668] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 20:30:51.016623] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 20:30:52.026726] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 20:30:52.032186] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 20:40:51.479298] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 20:40:52.489966] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 20:40:52.490446] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 20:50:52.043560] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 20:50:53.053966] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 20:50:53.054729] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 21:00:52.478524] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 21:00:53.496308] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 21:00:53.500431] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 21:10:19.726160] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2012-06-05 21:10:53.026093] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 21:10:54.036004] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 21:10:54.036434] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 21:20:53.602246] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 21:20:54.615420] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 21:20:54.617947] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 21:30:54.077244] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 21:30:55.085919] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 21:30:55.091006] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 21:40:54.546686] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 21:40:55.552464] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 21:40:55.553022] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 21:50:55.114735] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 21:50:56.124779] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 21:50:56.130301] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 22:00:55.694484] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 22:00:56.700196] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 22:00:56.700726] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 22:10:56.180551] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 22:10:57.198578] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 22:10:57.204404] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 22:20:56.737088] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 22:20:57.753094] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 22:20:57.757543] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 22:30:57.186287] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 22:30:58.196418] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 22:30:58.197010] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 22:40:57.661098] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 22:40:58.673894] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 22:40:58.680561] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 22:50:58.186509] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 22:50:59.196742] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 22:50:59.197188] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 23:00:58.769710] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 23:00:59.785623] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 23:00:59.791792] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 23:10:59.275857] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 23:11:00.286062] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 23:11:00.286838] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 23:20:59.852117] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 23:21:00.862451] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 23:21:00.862727] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 23:31:00.343437] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 23:31:01.357422] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 23:31:01.357899] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 23:41:00.877107] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 23:41:01.887471] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 23:41:01.887839] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-05 23:51:01.411227] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-05 23:51:02.421749] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-05 23:51:02.427431] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 00:01:01.956937] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 00:01:02.974690] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 00:01:02.977251] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 00:11:02.395533] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 00:11:03.407321] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 00:11:03.409388] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 00:21:03.007928] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 00:21:04.018010] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 00:21:04.027551] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 00:31:03.534833] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 00:31:04.545000] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 00:31:04.545630] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 00:41:04.096424] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 00:41:05.106468] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 00:41:05.111903] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 00:51:04.539277] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 00:51:05.549339] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 00:51:05.550001] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 01:01:05.100219] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 01:01:06.109697] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 01:01:06.113112] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 01:11:05.594815] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 01:11:06.612749] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 01:11:06.621994] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 01:21:06.093517] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 01:21:07.103651] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 01:21:07.104164] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 01:31:06.695001] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 01:31:07.705379] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 01:31:07.710824] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 01:41:07.232581] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 01:41:08.248433] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 01:41:08.248995] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 01:51:07.701672] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 01:51:08.711802] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 01:51:08.712128] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 02:01:08.210614] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-0: Stopping crawl for vol-client-1 , subvol went down >[2012-06-06 02:01:09.220636] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-1: Stopping crawl for vol-client-4 , subvol went down >[2012-06-06 02:01:09.226512] E [afr-self-heald.c:409:_crawl_proceed] 0-vol-replicate-2: Stopping crawl for vol-client-7 , subvol went down >[2012-06-06 02:05:31.465544] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3271ee570d] (-->/lib64/libpthread.so.0() [0x32722077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407c18]))) 0-: received signum (15), shutting down >[2012-06-06 02:05:32.489444] I [glusterfsd.c:1666:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.3.0qa45 >[2012-06-06 02:05:32.513859] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-2: adding option 'node-uuid' for volume 'vol-replicate-2' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d' >[2012-06-06 02:05:32.513921] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-1: adding option 'node-uuid' for volume 'vol-replicate-1' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d' >[2012-06-06 02:05:32.513959] I [graph.c:241:gf_add_cmdline_options] 0-vol-replicate-0: adding option 'node-uuid' for volume 'vol-replicate-0' with value '4c86a3c5-066c-4f30-8f15-762dfb29629d' >[2012-06-06 02:05:32.592965] I [client.c:2142:notify] 0-vol-client-0: parent translators are ready, attempting connect on transport >[2012-06-06 02:05:32.598093] I [client.c:2142:notify] 0-vol-client-1: parent translators are ready, attempting connect on transport >[2012-06-06 02:05:32.603362] I [client.c:2142:notify] 0-vol-client-2: parent translators are ready, attempting connect on transport >[2012-06-06 02:05:32.608304] I [client.c:2142:notify] 0-vol-client-3: parent translators are ready, attempting connect on transport >[2012-06-06 02:05:32.613078] I [client.c:2142:notify] 0-vol-client-4: parent translators are ready, attempting connect on transport >[2012-06-06 02:05:32.618376] I [client.c:2142:notify] 0-vol-client-5: parent translators are ready, attempting connect on transport >[2012-06-06 02:05:32.623345] I [client.c:2142:notify] 0-vol-client-6: parent translators are ready, attempting connect on transport >[2012-06-06 02:05:32.628233] I [client.c:2142:notify] 0-vol-client-7: parent translators are ready, attempting connect on transport >[2012-06-06 02:05:32.633249] I [client.c:2142:notify] 0-vol-client-8: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume vol-client-0 > 2: type protocol/client > 3: option remote-host 10.16.159.184 > 4: option remote-subvolume /export_b1/dir1 > 5: option transport-type tcp > 6: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 7: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 8: end-volume > 9: > 10: volume vol-client-1 > 11: type protocol/client > 12: option remote-host 10.16.159.188 > 13: option remote-subvolume /export_b1/dir1 > 14: option transport-type tcp > 15: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 16: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 17: end-volume > 18: > 19: volume vol-client-2 > 20: type protocol/client > 21: option remote-host 10.16.159.196 > 22: option remote-subvolume /export_b1/dir1 > 23: option transport-type tcp > 24: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 25: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 26: end-volume > 27: > 28: volume vol-client-3 > 29: type protocol/client > 30: option remote-host 10.16.159.184 > 31: option remote-subvolume /export_c1/dir1 > 32: option transport-type tcp > 33: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 34: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 35: end-volume > 36: > 37: volume vol-client-4 > 38: type protocol/client > 39: option remote-host 10.16.159.188 > 40: option remote-subvolume /export_c1/dir1 > 41: option transport-type tcp > 42: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 43: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 44: end-volume > 45: > 46: volume vol-client-5 > 47: type protocol/client > 48: option remote-host 10.16.159.196 > 49: option remote-subvolume /export_c1/dir1 > 50: option transport-type tcp > 51: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 52: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 53: end-volume > 54: > 55: volume vol-client-6 > 56: type protocol/client > 57: option remote-host 10.16.159.184 > 58: option remote-subvolume /export_d1/dir1 > 59: option transport-type tcp > 60: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 61: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 62: end-volume > 63: > 64: volume vol-client-7 > 65: type protocol/client > 66: option remote-host 10.16.159.188 > 67: option remote-subvolume /export_d1/dir1 > 68: option transport-type tcp > 69: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 70: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 71: end-volume > 72: > 73: volume vol-client-8 > 74: type protocol/client > 75: option remote-host 10.16.159.196 > 76: option remote-subvolume /export_d1/dir1 > 77: option transport-type tcp > 78: option username 0827907c-9b32-4c09-8b26-d9b2bb35cb52 > 79: option password 7f315b88-84e9-4167-ac2a-e1d07875dc3e > 80: end-volume > 81: > 82: volume vol-replicate-0 > 83: type cluster/replicate > 84: option background-self-heal-count 0 > 85: option metadata-self-heal on > 86: option data-self-heal on > 87: option entry-self-heal on > 88: option self-heal-daemon on > 89: option iam-self-heal-daemon yes > 90: subvolumes vol-client-0 vol-client-1 vol-client-2 > 91: end-volume > 92: > 93: volume vol-replicate-1 > 94: type cluster/replicate > 95: option background-self-heal-count 0 > 96: option metadata-self-heal on > 97: option data-self-heal on > 98: option entry-self-heal on > 99: option self-heal-daemon on >100: option iam-self-heal-daemon yes >101: subvolumes vol-client-3 vol-client-4 vol-client-5 >102: end-volume >103: >104: volume vol-replicate-2 >105: type cluster/replicate >106: option background-self-heal-count 0 >107: option metadata-self-heal on >108: option data-self-heal on >109: option entry-self-heal on >110: option self-heal-daemon on >111: option iam-self-heal-daemon yes >112: subvolumes vol-client-6 vol-client-7 vol-client-8 >113: end-volume >114: >115: volume glustershd >116: type debug/io-stats >117: subvolumes vol-replicate-0 vol-replicate-1 vol-replicate-2 >118: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-06 02:05:32.640966] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-0: changing port to 24009 (from 0) >[2012-06-06 02:05:32.641149] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-1: changing port to 24009 (from 0) >[2012-06-06 02:05:32.641308] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-2: changing port to 24009 (from 0) >[2012-06-06 02:05:32.641395] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-3: changing port to 24010 (from 0) >[2012-06-06 02:05:32.641473] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-4: changing port to 24010 (from 0) >[2012-06-06 02:05:32.641649] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-5: changing port to 24010 (from 0) >[2012-06-06 02:05:32.641734] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-6: changing port to 24011 (from 0) >[2012-06-06 02:05:32.641822] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-7: changing port to 24011 (from 0) >[2012-06-06 02:05:32.642407] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-vol-client-8: changing port to 24011 (from 0) >[2012-06-06 02:05:36.174004] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-06-06 02:05:36.174103] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-06-06 02:05:36.516990] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.517666] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-0: Connected to 10.16.159.184:24009, attached to remote volume '/export_b1/dir1'. >[2012-06-06 02:05:36.517714] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.517819] I [afr-common.c:3627:afr_notify] 0-vol-replicate-0: Subvolume 'vol-client-0' came back up; going online. >[2012-06-06 02:05:36.518253] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-0: Server lk version = 1 >[2012-06-06 02:05:36.521887] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.522243] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-1: Connected to 10.16.159.188:24009, attached to remote volume '/export_b1/dir1'. >[2012-06-06 02:05:36.522275] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.522861] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-1: Server lk version = 1 >[2012-06-06 02:05:36.527842] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.528453] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-2: Connected to 10.16.159.196:24009, attached to remote volume '/export_b1/dir1'. >[2012-06-06 02:05:36.528537] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.535155] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-2: Server lk version = 1 >[2012-06-06 02:05:36.536274] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.537446] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-3: Connected to 10.16.159.184:24010, attached to remote volume '/export_c1/dir1'. >[2012-06-06 02:05:36.537485] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.537767] I [afr-common.c:3627:afr_notify] 0-vol-replicate-1: Subvolume 'vol-client-3' came back up; going online. >[2012-06-06 02:05:36.539003] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-3: Server lk version = 1 >[2012-06-06 02:05:36.539732] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-4: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.540463] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-4: Connected to 10.16.159.188:24010, attached to remote volume '/export_c1/dir1'. >[2012-06-06 02:05:36.540499] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.541835] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-4: Server lk version = 1 >[2012-06-06 02:05:36.547427] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-5: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.548204] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-5: Connected to 10.16.159.196:24010, attached to remote volume '/export_c1/dir1'. >[2012-06-06 02:05:36.548243] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.552824] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-5: Server lk version = 1 >[2012-06-06 02:05:36.553418] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-6: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.554048] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-6: Connected to 10.16.159.184:24011, attached to remote volume '/export_d1/dir1'. >[2012-06-06 02:05:36.554079] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.554201] I [afr-common.c:3627:afr_notify] 0-vol-replicate-2: Subvolume 'vol-client-6' came back up; going online. >[2012-06-06 02:05:36.555330] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-6: Server lk version = 1 >[2012-06-06 02:05:36.557352] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-7: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.557697] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-7: Connected to 10.16.159.188:24011, attached to remote volume '/export_d1/dir1'. >[2012-06-06 02:05:36.557730] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.558864] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-7: Server lk version = 1 >[2012-06-06 02:05:36.563341] I [client-handshake.c:1636:select_server_supported_programs] 0-vol-client-8: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-06 02:05:36.563931] I [client-handshake.c:1433:client_setvolume_cbk] 0-vol-client-8: Connected to 10.16.159.196:24011, attached to remote volume '/export_d1/dir1'. >[2012-06-06 02:05:36.564005] I [client-handshake.c:1445:client_setvolume_cbk] 0-vol-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-06 02:05:36.564945] I [client-handshake.c:453:client_set_lk_version_cbk] 0-vol-client-8: Server lk version = 1 >pending frames: > >patchset: git://git.gluster.com/glusterfs.git >signal received: 11 >time of crash: 2012-06-06 02:12:08 >configuration details: >argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >st_atim.tv_nsec 1 >package-string: glusterfs 3.3.0qa45 >/lib64/libc.so.6[0x3271e32900] >/lib64/libc.so.6[0x3271f14f99] >/lib64/libc.so.6(xdr_u_int+0x53)[0x3271f141f3] >/lib64/libc.so.6(xdr_string+0x48)[0x3271f148e8] >/usr/local/lib/libgfxdr.so.0(xdr_gd1_mgmt_brick_op_req+0x29)[0x7f3af77c5c67] >/usr/local/lib/libgfxdr.so.0(xdr_to_generic+0x81)[0x7f3af77bd71f] >/usr/local/sbin/glusterfs(glusterfs_handle_translator_op+0x13e)[0x40a4b7] >/usr/local/lib/libglusterfs.so.0(synctask_wrap+0x38)[0x7f3af7c4a882] >/lib64/libc.so.6[0x3271e43610] >---------
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 829168
: 589744