Attachment 915661 Details for Bug 902953 – Comment

Comment

comment_5440508.txt (text/plain), 81.17 KB, created by John Morrissey on 2013-01-22 19:35:46 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: John Morrissey

Created: 2013-01-22 19:35:46 UTC

Size: 81.17 KB

patch

obsolete

>Description of problem: Filesystem is distributed across seven servers. After restarting glusterfsd on each server (one at a time, probably less than a minute in between, but might have restarted a couple close to simultaneously) to upgrade from 3.3.0 to 3.3.1, clients return ENOTCONN or EINVAL when trying to access the filesystem.
>
>The lead gluster thread seems to be spinning on trying to close the transport socket, even though the socket is already closed (strace output is below). The client log has these sporadically:
>
>[2013-01-22 13:46:05.756766] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>
>
>Version-Release number of selected component (if applicable): Clients are 3.3.1 on CentOS 5.
>
>
>How reproducible: This might involve restarting several servers; if I restart glusterfsd on a single server, clients seem fine.
>
>
>Steps to Reproduce:
>1. Restart glusterfsd on multiple servers in quick succession.
>  
>Actual results: Clients do not reconnect automatically to servers and return ENOTCONN or EINVAL upon I/O to the filesystem.
>
>
>Expected results: Clients should reconnect automatically after a short delay.
>
>
>Additional info:
>Process 7145 attached - interrupt to quit
>epoll_wait(3, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=9, u64=68719476745}}}, 257, 4294967295) = 1
>getsockopt(9, SOL_SOCKET, SO_ERROR, [384248733740564591], [4]) = 0
>shutdown(9, 2 /* send and receive */)   = -1 ENOTCONN (Transport endpoint is not connected)
>readv(9, [{"\0\0\0\0", 4}], 1)          = 0
>epoll_ctl(3, EPOLL_CTL_DEL, 9, NULL)    = 0
>close(9)                                = 0
>epoll_wait(3, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=9, u64=68719476745}}}, 257, 4294967295) = 1
>getsockopt(9, SOL_SOCKET, SO_ERROR, [379780868241096815], [4]) = 0
>shutdown(9, 2 /* send and receive */)   = -1 ENOTCONN (Transport endpoint is not connected)
>readv(9, [{"\0\0\0\0", 4}], 1)          = 0
>epoll_ctl(3, EPOLL_CTL_DEL, 9, NULL)    = 0
>close(9)                                = 0
>epoll_wait(3, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=9, u64=68719476745}}}, 257, 4294967295) = 1
>getsockopt(9, SOL_SOCKET, SO_ERROR, [424459523235774575], [4]) = 0
>shutdown(9, 2 /* send and receive */)   = -1 ENOTCONN (Transport endpoint is not connected)
>readv(9, [{"\0\0\0\0", 4}], 1)          = 0
>epoll_ctl(3, EPOLL_CTL_DEL, 9, NULL)    = 0
>close(9)                                = 0
>epoll_wait(3, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=9, u64=68719476745}}}, 257, 4294967295) = 1
>getsockopt(9, SOL_SOCKET, SO_ERROR, [419991657736306799], [4]) = 0
>shutdown(9, 2 /* send and receive */)   = -1 ENOTCONN (Transport endpoint is not connected)
>readv(9, [{"\0\0\0\0", 4}], 1)          = 0
>epoll_ctl(3, EPOLL_CTL_DEL, 9, NULL)    = 0
>close(9)                                = 0
>epoll_wait(3, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=9, u64=68719476745}}}, 257, 4294967295) = 1
>getsockopt(9, SOL_SOCKET, SO_ERROR, [411055926737371247], [4]) = 0
>shutdown(9, 2 /* send and receive */)   = -1 ENOTCONN (Transport endpoint is not connected)
>readv(9, [{"\0\0\0\0", 4}], 1)          = 0
>epoll_ctl(3, EPOLL_CTL_DEL, 9, NULL)    = 0
>close(9)                                = 0
>epoll_wait(3, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=9, u64=68719476745}}}, 257, 4294967295) = 1
>
>
>COMMAND    PID USER   FD   TYPE    DEVICE    SIZE      NODE NAME
>glusterfs 7145 root  cwd    DIR     253,0    4096         2 /
>glusterfs 7145 root  rtd    DIR     253,0    4096         2 /
>glusterfs 7145 root  txt    REG     253,0   80528   3342386 /usr/sbin/glusterfsd
>glusterfs 7145 root  mem    REG     253,0  473448     66563 /usr/lib64/libglusterfs.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   93200     66486 /usr/lib64/libgfrpc.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   85312     66482 /usr/lib64/libgfxdr.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0  143600   1016135 /lib64/ld-2.5.so
>glusterfs 7145 root  mem    REG     253,0 1722304   1016136 /lib64/libc-2.5.so
>glusterfs 7145 root  mem    REG     253,0   23360   1016138 /lib64/libdl-2.5.so
>glusterfs 7145 root  mem    REG     253,0  145824   1016139 /lib64/libpthread-2.5.so
>glusterfs 7145 root  mem    REG     253,0   85544   1016146 /lib64/libz.so.1.2.3
>glusterfs 7145 root  mem    REG     253,0 1366368   1016147 /lib64/libcrypto.so.0.9.8e
>glusterfs 7145 root  mem    REG     253,0   78904   2524138 /usr/lib64/glusterfs/3.3.1/rpc-transport/socket.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   53880   1015835 /lib64/libnss_files-2.5.so
>glusterfs 7145 root  mem    REG     253,0  265736   2720153 /usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0  291504   2524146 /usr/lib64/glusterfs/3.3.1/xlator/cluster/dht.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   68904   2720149 /usr/lib64/glusterfs/3.3.1/xlator/performance/write-behind.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   42480   2720145 /usr/lib64/glusterfs/3.3.1/xlator/performance/read-ahead.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   70888   2720133 /usr/lib64/glusterfs/3.3.1/xlator/performance/io-cache.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   90008   2720142 /usr/lib64/glusterfs/3.3.1/xlator/performance/quick-read.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   45424   2720139 /usr/lib64/glusterfs/3.3.1/xlator/performance/md-cache.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   94744   2624966 /usr/lib64/glusterfs/3.3.1/xlator/debug/io-stats.so.0.0.0
>glusterfs 7145 root  mem    REG     253,0   58272   2720157 /usr/lib64/glusterfs/3.3.1/xlator/system/posix-acl.so.0.0.0
>glusterfs 7145 root  DEL    REG     253,0            689434 /var/run/nscd/dbcKYInY
>glusterfs 7145 root  mem    REG     253,0  150424   2720225 /usr/lib64/glusterfs/3.3.1/xlator/mount/fuse.so.0.0.0
>glusterfs 7145 root    0r   CHR       1,3              6198 /dev/null
>glusterfs 7145 root    1w   CHR       1,3              6198 /dev/null
>glusterfs 7145 root    2w   CHR       1,3              6198 /dev/null
>glusterfs 7145 root    3r  0000      0,11       0 475393280 eventpoll
>glusterfs 7145 root    4w   REG     253,0 1692383    721090 /var/log/glusterfs/n-hsphS10-hsph_gluster.log
>glusterfs 7145 root    5r   CHR       1,9              3595 /dev/urandom
>glusterfs 7145 root    6u  IPv4 475600463               TCP hsph02.rc.fas.harvard.edu:1020->hsphgoss01.rc.fas.harvard.edu:24007 (ESTABLISHED)
>glusterfs 7145 root    7u   CHR    10,229              6210 /dev/fuse
>glusterfs 7145 root    8u  IPv4 475600698               TCP hsph02.rc.fas.harvard.edu:1019->hsphgoss01.rc.fas.harvard.edu:24014 (ESTABLISHED)
>glusterfs 7145 root   10u   REG     253,0       0    369125 /tmp/tmpfpfRS6C (deleted)
>glusterfs 7145 root   11u  IPv4 475600700               TCP hsph02.rc.fas.harvard.edu:1018->hsphgoss01.rc.fas.harvard.edu:24023 (ESTABLISHED)
>glusterfs 7145 root   18u  IPv4 475600702               TCP hsph02.rc.fas.harvard.edu:1017->hsphgoss01.rc.fas.harvard.edu:24022 (ESTABLISHED)
>glusterfs 7145 root   25u  IPv4 475600704               TCP hsph02.rc.fas.harvard.edu:1016->hsphgoss01.rc.fas.harvard.edu:24020 (ESTABLISHED)
>glusterfs 7145 root   32u  IPv4 475600706               TCP hsph02.rc.fas.harvard.edu:1015->hsphgoss01.rc.fas.harvard.edu:24013 (ESTABLISHED)
>glusterfs 7145 root   39u  IPv4 475600710               TCP hsph02.rc.fas.harvard.edu:1014->hsphgoss01.rc.fas.harvard.edu:24015 (ESTABLISHED)
>glusterfs 7145 root   46u  IPv4 475600712               TCP hsph02.rc.fas.harvard.edu:1013->hsphgoss01.rc.fas.harvard.edu:24021 (ESTABLISHED)
>glusterfs 7145 root   53u  IPv4 475600714               TCP hsph02.rc.fas.harvard.edu:1012->hsphgoss01.rc.fas.harvard.edu:24017 (ESTABLISHED)
>glusterfs 7145 root   60u  IPv4 475600716               TCP hsph02.rc.fas.harvard.edu:1011->hsphgoss01.rc.fas.harvard.edu:24024 (ESTABLISHED)
>glusterfs 7145 root   67u  IPv4 475600718               TCP hsph02.rc.fas.harvard.edu:surf->hsphgoss01.rc.fas.harvard.edu:24018 (ESTABLISHED)
>glusterfs 7145 root   74u  IPv4 475600720               TCP hsph02.rc.fas.harvard.edu:1007->hsphgoss01.rc.fas.harvard.edu:24019 (ESTABLISHED)
>glusterfs 7145 root   81u  IPv4 475600722               TCP hsph02.rc.fas.harvard.edu:1006->hsphgoss01.rc.fas.harvard.edu:24011 (ESTABLISHED)
>glusterfs 7145 root   88u  IPv4 475600725               TCP hsph02.rc.fas.harvard.edu:1005->hsphgoss01.rc.fas.harvard.edu:24016 (ESTABLISHED)
>glusterfs 7145 root   95u  IPv4 475600727               TCP hsph02.rc.fas.harvard.edu:1004->hsphgoss01.rc.fas.harvard.edu:24012 (ESTABLISHED)
>glusterfs 7145 root  102u  IPv4 475600729               TCP hsph02.rc.fas.harvard.edu:1003->hsphgoss01.rc.fas.harvard.edu:24025 (ESTABLISHED)
>
>[2013-01-22 13:00:43.406830] W [socket.c:1512:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24007)
>[2013-01-22 13:00:43.607994] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-28: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24014)
>[2013-01-22 13:00:43.608135] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-28: disconnected
>[2013-01-22 13:00:43.609032] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-91: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24023)
>[2013-01-22 13:00:43.609084] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-91: disconnected
>[2013-01-22 13:00:43.609442] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-84: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24022)
>[2013-01-22 13:00:43.609506] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-84: disconnected
>[2013-01-22 13:00:43.609540] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-70: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24020)
>[2013-01-22 13:00:43.609584] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-70: disconnected
>[2013-01-22 13:00:43.610351] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-21: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24013)
>[2013-01-22 13:00:43.610400] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-21: disconnected
>[2013-01-22 13:00:43.610448] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24010)
>[2013-01-22 13:00:43.610497] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-0: disconnected
>[2013-01-22 13:00:43.610572] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-56: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24018)
>[2013-01-22 13:00:43.610636] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-56: disconnected
>[2013-01-22 13:00:43.610692] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-35: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24015)
>[2013-01-22 13:00:43.610733] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-35: disconnected
>[2013-01-22 13:00:43.610762] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-49: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24017)
>[2013-01-22 13:00:43.610801] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-49: disconnected
>[2013-01-22 13:00:43.610829] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-77: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24021)
>[2013-01-22 13:00:43.610867] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-77: disconnected
>[2013-01-22 13:00:43.610899] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-98: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24024)
>[2013-01-22 13:00:43.610941] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-98: disconnected
>[2013-01-22 13:00:43.610973] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-63: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24019)
>[2013-01-22 13:00:43.611013] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-63: disconnected
>[2013-01-22 13:00:43.611042] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-7: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24011)
>[2013-01-22 13:00:43.611081] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-7: disconnected
>[2013-01-22 13:00:43.611111] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-42: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24016)
>[2013-01-22 13:00:43.611165] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-42: disconnected
>[2013-01-22 13:00:43.611205] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-14: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24012)
>[2013-01-22 13:00:43.611245] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-14: disconnected
>[2013-01-22 13:00:43.611273] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-105: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.96:24025)
>[2013-01-22 13:00:43.611328] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-105: disconnected
>[2013-01-22 13:00:53.932863] E [socket.c:1715:socket_connect_finish] 0-glusterfs: connection to 10.242.62.96:24007 failed (Connection refused)
>[2013-01-22 13:00:53.937337] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-28: connection to 10.242.62.96:24014 failed (Connection refused)
>[2013-01-22 13:00:53.941206] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-91: connection to 10.242.62.96:24023 failed (Connection refused)
>[2013-01-22 13:00:53.945306] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-84: connection to 10.242.62.96:24022 failed (Connection refused)
>[2013-01-22 13:00:53.949971] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-70: connection to 10.242.62.96:24020 failed (Connection refused)
>[2013-01-22 13:00:53.954416] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-21: connection to 10.242.62.96:24013 failed (Connection refused)
>[2013-01-22 13:00:53.958882] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-0: connection to 10.242.62.96:24010 failed (Connection refused)
>[2013-01-22 13:00:53.963540] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-56: connection to 10.242.62.96:24018 failed (Connection refused)
>[2013-01-22 13:00:53.967943] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-35: connection to 10.242.62.96:24015 failed (Connection refused)
>[2013-01-22 13:00:53.972535] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-49: connection to 10.242.62.96:24017 failed (Connection refused)
>[2013-01-22 13:00:53.976665] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-77: connection to 10.242.62.96:24021 failed (Connection refused)
>[2013-01-22 13:00:53.981202] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-98: connection to 10.242.62.96:24024 failed (Connection refused)
>[2013-01-22 13:00:53.986310] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-63: connection to 10.242.62.96:24019 failed (Connection refused)
>[2013-01-22 13:00:53.990617] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-7: connection to 10.242.62.96:24011 failed (Connection refused)
>[2013-01-22 13:00:53.995354] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-42: connection to 10.242.62.96:24016 failed (Connection refused)
>[2013-01-22 13:00:53.999825] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-14: connection to 10.242.62.96:24012 failed (Connection refused)
>[2013-01-22 13:00:54.004958] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-105: connection to 10.242.62.96:24025 failed (Connection refused)
>[2013-01-22 13:02:41.721608] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-01-22 13:02:46.380801] I [glusterfsd-mgmt.c:1569:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2013-01-22 13:02:47.837541] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-28: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.837889] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-28: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.837921] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-28: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.837940] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-28: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo05" is not found
>[2013-01-22 13:02:47.837958] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-28: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.841618] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-91: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.841973] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-91: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.842002] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-91: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.842022] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-91: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo14" is not found
>[2013-01-22 13:02:47.842040] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-91: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.845451] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-84: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.845744] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-84: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.845774] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-84: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.845794] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-84: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo13" is not found
>[2013-01-22 13:02:47.845814] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-84: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.849445] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-70: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.849772] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-70: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.849803] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-70: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.849825] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-70: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo11" is not found
>[2013-01-22 13:02:47.849853] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-70: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.853964] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-21: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.854272] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-21: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.854299] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-21: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.854318] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-21: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo04" is not found
>[2013-01-22 13:02:47.854342] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-21: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.862427] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-56: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.862780] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-56: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.862810] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-56: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.862828] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-56: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo09" is not found
>[2013-01-22 13:02:47.862857] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-56: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.866711] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-35: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.867052] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-35: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.867085] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-35: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.867108] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-35: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo06" is not found
>[2013-01-22 13:02:47.867128] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-35: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.870684] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-49: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.870977] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-49: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.871007] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-49: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.871028] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-49: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo08" is not found
>[2013-01-22 13:02:47.871048] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-49: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.874656] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-77: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.874977] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-77: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.875005] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-77: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.875024] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-77: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo12" is not found
>[2013-01-22 13:02:47.875042] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-77: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.878501] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-98: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.878810] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-98: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.878839] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-98: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.878862] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-98: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo15" is not found
>[2013-01-22 13:02:47.878881] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-98: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.882490] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-63: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.882839] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-63: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.882879] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-63: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.882903] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-63: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo10" is not found
>[2013-01-22 13:02:47.882926] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-63: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.886368] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-7: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.886729] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-7: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.886765] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-7: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.886788] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-7: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo02" is not found
>[2013-01-22 13:02:47.886808] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-7: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.889995] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-42: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.890337] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-42: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.890365] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-42: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.890383] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-42: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo07" is not found
>[2013-01-22 13:02:47.890400] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-42: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.893671] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-14: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.894030] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-14: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.894057] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-14: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.894075] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-14: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo03" is not found
>[2013-01-22 13:02:47.894094] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-14: sending CHILD_CONNECTING event
>[2013-01-22 13:02:47.897814] I [client-handshake.c:1636:select_server_supported_programs] 0-hsph_gluster-client-105: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
>[2013-01-22 13:02:47.898152] W [client-handshake.c:1342:client_setvolume_cbk] 0-hsph_gluster-client-105: failed to set the volume (No such file or directory)
>[2013-01-22 13:02:47.898182] W [client-handshake.c:1368:client_setvolume_cbk] 0-hsph_gluster-client-105: failed to get 'process-uuid' from reply dict
>[2013-01-22 13:02:47.898204] E [client-handshake.c:1374:client_setvolume_cbk] 0-hsph_gluster-client-105: SETVOLUME on remote-host failed: remote-subvolume "/gluster_expo16" is not found
>[2013-01-22 13:02:47.898224] I [client-handshake.c:1474:client_setvolume_cbk] 0-hsph_gluster-client-105: sending CHILD_CONNECTING event
>[2013-01-22 13:04:19.222804] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-8: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24011)
>[2013-01-22 13:04:19.222963] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-8: disconnected
>[2013-01-22 13:04:19.223043] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-92: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24023)
>[2013-01-22 13:04:19.223088] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-92: disconnected
>[2013-01-22 13:04:19.223967] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-78: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24021)
>[2013-01-22 13:04:19.224024] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-78: disconnected
>[2013-01-22 13:04:19.224063] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-57: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24018)
>[2013-01-22 13:04:19.224125] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-57: disconnected
>[2013-01-22 13:04:19.224283] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-15: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24012)
>[2013-01-22 13:04:19.224338] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-15: disconnected
>[2013-01-22 13:04:19.224582] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-71: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24020)
>[2013-01-22 13:04:19.224636] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-71: disconnected
>[2013-01-22 13:04:19.224674] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-106: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24025)
>[2013-01-22 13:04:19.224718] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-106: disconnected
>[2013-01-22 13:04:19.226078] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-36: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24015)
>[2013-01-22 13:04:19.226155] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-36: disconnected
>[2013-01-22 13:04:19.226210] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-43: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24016)
>[2013-01-22 13:04:19.226334] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-43: disconnected
>[2013-01-22 13:04:19.226365] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-50: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24017)
>[2013-01-22 13:04:19.226403] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-50: disconnected
>[2013-01-22 13:04:19.226445] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-99: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24024)
>[2013-01-22 13:04:19.226487] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-99: disconnected
>[2013-01-22 13:04:19.226613] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-22: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24013)
>[2013-01-22 13:04:19.226657] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-22: disconnected
>[2013-01-22 13:04:19.226688] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-85: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24022)
>[2013-01-22 13:04:19.226728] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-85: disconnected
>[2013-01-22 13:04:19.227100] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-29: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24014)
>[2013-01-22 13:04:19.227145] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-29: disconnected
>[2013-01-22 13:04:19.228075] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-64: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24019)
>[2013-01-22 13:04:19.228129] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-64: disconnected
>[2013-01-22 13:04:19.229623] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.97:24010)
>[2013-01-22 13:04:19.229704] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-1: disconnected
>[2013-01-22 13:04:30.160007] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-8: connection to 10.242.62.97:24011 failed (Connection refused)
>[2013-01-22 13:04:30.165183] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-92: connection to 10.242.62.97:24023 failed (Connection refused)
>[2013-01-22 13:04:30.169075] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-78: connection to 10.242.62.97:24021 failed (Connection refused)
>[2013-01-22 13:04:30.173808] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-57: connection to 10.242.62.97:24018 failed (Connection refused)
>[2013-01-22 13:04:30.178217] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-15: connection to 10.242.62.97:24012 failed (Connection refused)
>[2013-01-22 13:04:30.182845] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-71: connection to 10.242.62.97:24020 failed (Connection refused)
>[2013-01-22 13:04:30.187614] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-106: connection to 10.242.62.97:24025 failed (Connection refused)
>[2013-01-22 13:04:30.191868] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-36: connection to 10.242.62.97:24015 failed (Connection refused)
>[2013-01-22 13:04:30.196723] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-43: connection to 10.242.62.97:24016 failed (Connection refused)
>[2013-01-22 13:04:30.202367] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-50: connection to 10.242.62.97:24017 failed (Connection refused)
>[2013-01-22 13:04:30.207349] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-99: connection to 10.242.62.97:24024 failed (Connection refused)
>[2013-01-22 13:04:30.211269] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-22: connection to 10.242.62.97:24013 failed (Connection refused)
>[2013-01-22 13:04:30.216424] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-85: connection to 10.242.62.97:24022 failed (Connection refused)
>[2013-01-22 13:04:30.221131] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-29: connection to 10.242.62.97:24014 failed (Connection refused)
>[2013-01-22 13:04:30.225658] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-64: connection to 10.242.62.97:24019 failed (Connection refused)
>[2013-01-22 13:04:30.230938] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-1: connection to 10.242.62.97:24010 failed (Connection refused)
>[2013-01-22 13:07:38.092585] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-01-22 13:07:44.235959] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-01-22 13:09:04.481880] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-16: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24012)
>[2013-01-22 13:09:04.482033] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-16: disconnected
>[2013-01-22 13:09:04.482108] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-23: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24013)
>[2013-01-22 13:09:04.482154] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-23: disconnected
>[2013-01-22 13:09:04.482307] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-9: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24011)
>[2013-01-22 13:09:04.482358] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-9: disconnected
>[2013-01-22 13:09:04.482393] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-93: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24023)
>[2013-01-22 13:09:04.482436] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-93: disconnected
>[2013-01-22 13:09:04.482702] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-72: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24020)
>[2013-01-22 13:09:04.482747] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-72: disconnected
>[2013-01-22 13:09:04.482784] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24010)
>[2013-01-22 13:09:04.482825] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-2: disconnected
>[2013-01-22 13:09:04.483413] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-37: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24015)
>[2013-01-22 13:09:04.483459] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-37: disconnected
>[2013-01-22 13:09:04.483495] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-100: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24024)
>[2013-01-22 13:09:04.483536] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-100: disconnected
>[2013-01-22 13:09:04.483875] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-65: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24019)
>[2013-01-22 13:09:04.483920] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-65: disconnected
>[2013-01-22 13:09:04.484061] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-79: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24021)
>[2013-01-22 13:09:04.484107] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-79: disconnected
>[2013-01-22 13:09:04.484360] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-107: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24025)
>[2013-01-22 13:09:04.484447] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-107: disconnected
>[2013-01-22 13:09:04.484477] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-58: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24018)
>[2013-01-22 13:09:04.484516] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-58: disconnected
>[2013-01-22 13:09:04.484542] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-51: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24017)
>[2013-01-22 13:09:04.484580] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-51: disconnected
>[2013-01-22 13:09:04.484606] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-30: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24014)
>[2013-01-22 13:09:04.484643] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-30: disconnected
>[2013-01-22 13:09:04.484681] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-44: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24016)
>[2013-01-22 13:09:04.484731] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-44: disconnected
>[2013-01-22 13:09:04.484807] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-86: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.98:24022)
>[2013-01-22 13:09:04.484849] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-86: disconnected
>[2013-01-22 13:09:14.631179] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-16: connection to 10.242.62.98:24012 failed (Connection refused)
>[2013-01-22 13:09:14.636040] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-23: connection to 10.242.62.98:24013 failed (Connection refused)
>[2013-01-22 13:09:14.640185] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-9: connection to 10.242.62.98:24011 failed (Connection refused)
>[2013-01-22 13:09:14.644953] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-93: connection to 10.242.62.98:24023 failed (Connection refused)
>[2013-01-22 13:09:14.650007] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-72: connection to 10.242.62.98:24020 failed (Connection refused)
>[2013-01-22 13:09:14.654362] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-2: connection to 10.242.62.98:24010 failed (Connection refused)
>[2013-01-22 13:09:14.659691] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-37: connection to 10.242.62.98:24015 failed (Connection refused)
>[2013-01-22 13:09:14.665021] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-100: connection to 10.242.62.98:24024 failed (Connection refused)
>[2013-01-22 13:09:14.669770] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-65: connection to 10.242.62.98:24019 failed (Connection refused)
>[2013-01-22 13:09:14.674469] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-79: connection to 10.242.62.98:24021 failed (Connection refused)
>[2013-01-22 13:09:14.679304] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-107: connection to 10.242.62.98:24025 failed (Connection refused)
>[2013-01-22 13:09:14.683933] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-58: connection to 10.242.62.98:24018 failed (Connection refused)
>[2013-01-22 13:09:14.689176] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-51: connection to 10.242.62.98:24017 failed (Connection refused)
>[2013-01-22 13:09:14.693145] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-30: connection to 10.242.62.98:24014 failed (Connection refused)
>[2013-01-22 13:09:14.698501] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-44: connection to 10.242.62.98:24016 failed (Connection refused)
>[2013-01-22 13:09:14.702515] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-86: connection to 10.242.62.98:24022 failed (Connection refused)
>[2013-01-22 13:11:40.631050] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-59: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24018)
>[2013-01-22 13:11:40.631184] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-59: disconnected
>[2013-01-22 13:11:40.631665] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-66: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24019)
>[2013-01-22 13:11:40.631709] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-66: disconnected
>[2013-01-22 13:11:40.631762] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-73: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24020)
>[2013-01-22 13:11:40.631800] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-73: disconnected
>[2013-01-22 13:11:40.631831] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-52: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24017)
>[2013-01-22 13:11:40.631869] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-52: disconnected
>[2013-01-22 13:11:40.631899] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-80: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24021)
>[2013-01-22 13:11:40.631935] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-80: disconnected
>[2013-01-22 13:11:40.632041] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-38: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24015)
>[2013-01-22 13:11:40.632083] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-38: disconnected
>[2013-01-22 13:11:40.632421] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-10: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24011)
>[2013-01-22 13:11:40.632464] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-10: disconnected
>[2013-01-22 13:11:40.632491] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-87: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24022)
>[2013-01-22 13:11:40.632525] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-87: disconnected
>[2013-01-22 13:11:40.632550] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-31: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24014)
>[2013-01-22 13:11:40.632597] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-31: disconnected
>[2013-01-22 13:11:40.632625] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-108: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24025)
>[2013-01-22 13:11:40.632671] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-108: disconnected
>[2013-01-22 13:11:40.632801] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-94: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24023)
>[2013-01-22 13:11:40.632847] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-94: disconnected
>[2013-01-22 13:11:40.632877] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24010)
>[2013-01-22 13:11:40.632922] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-3: disconnected
>[2013-01-22 13:11:40.632948] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-101: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24024)
>[2013-01-22 13:11:40.632986] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-101: disconnected
>[2013-01-22 13:11:40.633013] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-24: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24013)
>[2013-01-22 13:11:40.633086] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-24: disconnected
>[2013-01-22 13:11:40.633896] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-45: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24016)
>[2013-01-22 13:11:40.633933] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-45: disconnected
>[2013-01-22 13:11:40.634027] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-17: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.99:24012)
>[2013-01-22 13:11:40.634061] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-17: disconnected
>[2013-01-22 13:11:51.068105] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-59: connection to 10.242.62.99:24018 failed (Connection refused)
>[2013-01-22 13:11:51.073125] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-66: connection to 10.242.62.99:24019 failed (Connection refused)
>[2013-01-22 13:11:51.077395] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-73: connection to 10.242.62.99:24020 failed (Connection refused)
>[2013-01-22 13:11:51.081870] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-52: connection to 10.242.62.99:24017 failed (Connection refused)
>[2013-01-22 13:11:51.086444] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-80: connection to 10.242.62.99:24021 failed (Connection refused)
>[2013-01-22 13:11:51.090820] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-38: connection to 10.242.62.99:24015 failed (Connection refused)
>[2013-01-22 13:11:51.095409] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-10: connection to 10.242.62.99:24011 failed (Connection refused)
>[2013-01-22 13:11:51.099949] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-87: connection to 10.242.62.99:24022 failed (Connection refused)
>[2013-01-22 13:11:51.104825] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-31: connection to 10.242.62.99:24014 failed (Connection refused)
>[2013-01-22 13:11:51.109305] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-108: connection to 10.242.62.99:24025 failed (Connection refused)
>[2013-01-22 13:11:51.114356] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-94: connection to 10.242.62.99:24023 failed (Connection refused)
>[2013-01-22 13:11:51.118904] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-3: connection to 10.242.62.99:24010 failed (Connection refused)
>[2013-01-22 13:11:51.123647] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-101: connection to 10.242.62.99:24024 failed (Connection refused)
>[2013-01-22 13:11:51.128477] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-24: connection to 10.242.62.99:24013 failed (Connection refused)
>[2013-01-22 13:11:51.133138] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-45: connection to 10.242.62.99:24016 failed (Connection refused)
>[2013-01-22 13:11:51.137962] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-17: connection to 10.242.62.99:24012 failed (Connection refused)
>[2013-01-22 13:11:52.804920] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-81: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24021)
>[2013-01-22 13:11:52.805047] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-81: disconnected
>[2013-01-22 13:11:52.805376] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-32: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24014)
>[2013-01-22 13:11:52.805422] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-32: disconnected
>[2013-01-22 13:11:52.805609] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-109: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24025)
>[2013-01-22 13:11:52.805683] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-109: disconnected
>[2013-01-22 13:11:52.805717] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-4: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24010)
>[2013-01-22 13:11:52.805765] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-4: disconnected
>[2013-01-22 13:11:52.805794] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-67: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24019)
>[2013-01-22 13:11:52.805832] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-67: disconnected
>[2013-01-22 13:11:52.805862] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-74: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24020)
>[2013-01-22 13:11:52.805914] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-74: disconnected
>[2013-01-22 13:11:52.805948] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-60: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24018)
>[2013-01-22 13:11:52.805988] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-60: disconnected
>[2013-01-22 13:11:52.806280] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-39: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24015)
>[2013-01-22 13:11:52.806320] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-39: disconnected
>[2013-01-22 13:11:52.806348] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-46: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24016)
>[2013-01-22 13:11:52.806385] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-46: disconnected
>[2013-01-22 13:11:52.806422] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-25: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24013)
>[2013-01-22 13:11:52.806458] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-25: disconnected
>[2013-01-22 13:11:52.806488] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-102: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24024)
>[2013-01-22 13:11:52.806525] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-102: disconnected
>[2013-01-22 13:11:52.806629] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-53: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24017)
>[2013-01-22 13:11:52.806664] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-53: disconnected
>[2013-01-22 13:11:52.806963] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-11: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24011)
>[2013-01-22 13:11:52.807017] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-11: disconnected
>[2013-01-22 13:11:52.807047] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-88: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24022)
>[2013-01-22 13:11:52.807082] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-88: disconnected
>[2013-01-22 13:11:52.807107] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-18: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24012)
>[2013-01-22 13:11:52.807153] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-18: disconnected
>[2013-01-22 13:11:52.807284] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-95: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.100:24023)
>[2013-01-22 13:11:52.807378] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-95: disconnected
>[2013-01-22 13:12:00.028296] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-96: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24023)
>[2013-01-22 13:12:00.028418] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-96: disconnected
>[2013-01-22 13:12:00.035958] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-103: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24024)
>[2013-01-22 13:12:00.036020] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-103: disconnected
>[2013-01-22 13:12:00.036140] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-33: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24014)
>[2013-01-22 13:12:00.036195] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-33: disconnected
>[2013-01-22 13:12:00.036235] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-54: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24017)
>[2013-01-22 13:12:00.036287] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-54: disconnected
>[2013-01-22 13:12:00.036313] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-26: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24013)
>[2013-01-22 13:12:00.036360] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-26: disconnected
>[2013-01-22 13:12:00.036388] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-82: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24021)
>[2013-01-22 13:12:00.036427] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-82: disconnected
>[2013-01-22 13:12:00.036453] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-40: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24015)
>[2013-01-22 13:12:00.036500] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-40: disconnected
>[2013-01-22 13:12:00.036536] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-75: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24020)
>[2013-01-22 13:12:00.036570] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-75: disconnected
>[2013-01-22 13:12:00.036594] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-68: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24019)
>[2013-01-22 13:12:00.036631] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-68: disconnected
>[2013-01-22 13:12:00.036658] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-47: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24016)
>[2013-01-22 13:12:00.036691] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-47: disconnected
>[2013-01-22 13:12:00.036717] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-19: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24012)
>[2013-01-22 13:12:00.036754] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-19: disconnected
>[2013-01-22 13:12:00.036892] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-110: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24025)
>[2013-01-22 13:12:00.036934] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-110: disconnected
>[2013-01-22 13:12:00.036978] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-61: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24018)
>[2013-01-22 13:12:00.037030] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-61: disconnected
>[2013-01-22 13:12:00.037058] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-89: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24022)
>[2013-01-22 13:12:00.037121] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-89: disconnected
>[2013-01-22 13:12:00.037152] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-5: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24010)
>[2013-01-22 13:12:00.037190] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-5: disconnected
>[2013-01-22 13:12:00.037221] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-12: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.101:24011)
>[2013-01-22 13:12:00.037257] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-12: disconnected
>[2013-01-22 13:12:02.983233] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-81: connection to 10.242.62.100:24021 failed (Connection refused)
>[2013-01-22 13:12:02.988271] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-32: connection to 10.242.62.100:24014 failed (Connection refused)
>[2013-01-22 13:12:02.992958] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-109: connection to 10.242.62.100:24025 failed (Connection refused)
>[2013-01-22 13:12:02.997513] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-4: connection to 10.242.62.100:24010 failed (Connection refused)
>[2013-01-22 13:12:03.002624] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-67: connection to 10.242.62.100:24019 failed (Connection refused)
>[2013-01-22 13:12:03.007018] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-74: connection to 10.242.62.100:24020 failed (Connection refused)
>[2013-01-22 13:12:03.011223] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-60: connection to 10.242.62.100:24018 failed (Connection refused)
>[2013-01-22 13:12:03.015725] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-39: connection to 10.242.62.100:24015 failed (Connection refused)
>[2013-01-22 13:12:03.020021] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-46: connection to 10.242.62.100:24016 failed (Connection refused)
>[2013-01-22 13:12:03.024629] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-25: connection to 10.242.62.100:24013 failed (Connection refused)
>[2013-01-22 13:12:03.028887] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-102: connection to 10.242.62.100:24024 failed (Connection refused)
>[2013-01-22 13:12:03.033405] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-53: connection to 10.242.62.100:24017 failed (Connection refused)
>[2013-01-22 13:12:03.037942] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-11: connection to 10.242.62.100:24011 failed (Connection refused)
>[2013-01-22 13:12:03.042864] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-88: connection to 10.242.62.100:24022 failed (Connection refused)
>[2013-01-22 13:12:03.047489] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-18: connection to 10.242.62.100:24012 failed (Connection refused)
>[2013-01-22 13:12:03.052190] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-95: connection to 10.242.62.100:24023 failed (Connection refused)
>[2013-01-22 13:12:10.663064] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-96: connection to 10.242.62.101:24023 failed (Connection refused)
>[2013-01-22 13:12:10.667597] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-103: connection to 10.242.62.101:24024 failed (Connection refused)
>[2013-01-22 13:12:10.672124] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-33: connection to 10.242.62.101:24014 failed (Connection refused)
>[2013-01-22 13:12:10.676593] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-54: connection to 10.242.62.101:24017 failed (Connection refused)
>[2013-01-22 13:12:10.681284] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-26: connection to 10.242.62.101:24013 failed (Connection refused)
>[2013-01-22 13:12:10.685453] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-82: connection to 10.242.62.101:24021 failed (Connection refused)
>[2013-01-22 13:12:10.690317] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-40: connection to 10.242.62.101:24015 failed (Connection refused)
>[2013-01-22 13:12:10.694327] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-75: connection to 10.242.62.101:24020 failed (Connection refused)
>[2013-01-22 13:12:10.699470] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-68: connection to 10.242.62.101:24019 failed (Connection refused)
>[2013-01-22 13:12:10.703813] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-47: connection to 10.242.62.101:24016 failed (Connection refused)
>[2013-01-22 13:12:10.708850] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-19: connection to 10.242.62.101:24012 failed (Connection refused)
>[2013-01-22 13:12:10.713507] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-110: connection to 10.242.62.101:24025 failed (Connection refused)
>[2013-01-22 13:12:10.718454] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-61: connection to 10.242.62.101:24018 failed (Connection refused)
>[2013-01-22 13:12:10.724021] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-89: connection to 10.242.62.101:24022 failed (Connection refused)
>[2013-01-22 13:12:10.728495] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-5: connection to 10.242.62.101:24010 failed (Connection refused)
>[2013-01-22 13:12:10.733049] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-12: connection to 10.242.62.101:24011 failed (Connection refused)
>[2013-01-22 13:12:11.256372] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-41: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24015)
>[2013-01-22 13:12:11.256496] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-41: disconnected
>[2013-01-22 13:12:11.256549] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-27: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24013)
>[2013-01-22 13:12:11.256597] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-27: disconnected
>[2013-01-22 13:12:11.256869] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-6: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24010)
>[2013-01-22 13:12:11.256911] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-6: disconnected
>[2013-01-22 13:12:11.257239] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-62: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24018)
>[2013-01-22 13:12:11.257285] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-62: disconnected
>[2013-01-22 13:12:11.257656] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-20: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24012)
>[2013-01-22 13:12:11.257704] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-20: disconnected
>[2013-01-22 13:12:11.257911] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-13: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24011)
>[2013-01-22 13:12:11.257956] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-13: disconnected
>[2013-01-22 13:12:11.257988] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-48: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24016)
>[2013-01-22 13:12:11.258042] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-48: disconnected
>[2013-01-22 13:12:11.258179] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-55: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24017)
>[2013-01-22 13:12:11.258240] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-55: disconnected
>[2013-01-22 13:12:11.258497] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-34: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24014)
>[2013-01-22 13:12:11.258537] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-34: disconnected
>[2013-01-22 13:12:11.266503] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-111: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24025)
>[2013-01-22 13:12:11.266565] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-111: disconnected
>[2013-01-22 13:12:11.266599] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-83: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24021)
>[2013-01-22 13:12:11.266639] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-83: disconnected
>[2013-01-22 13:12:11.266667] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-90: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24022)
>[2013-01-22 13:12:11.266705] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-90: disconnected
>[2013-01-22 13:12:11.266742] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-97: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24023)
>[2013-01-22 13:12:11.266797] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-97: disconnected
>[2013-01-22 13:12:11.266839] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-76: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24020)
>[2013-01-22 13:12:11.266878] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-76: disconnected
>[2013-01-22 13:12:11.266904] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-69: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24019)
>[2013-01-22 13:12:11.266940] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-69: disconnected
>[2013-01-22 13:12:11.267619] W [socket.c:1512:__socket_proto_state_machine] 0-hsph_gluster-client-104: reading from socket failed. Error (Transport endpoint is not connected), peer (10.242.62.102:24024)
>[2013-01-22 13:12:11.267673] I [client.c:2090:client_rpc_notify] 0-hsph_gluster-client-104: disconnected
>[2013-01-22 13:12:21.984176] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-41: connection to 10.242.62.102:24015 failed (Connection refused)
>[2013-01-22 13:12:21.988736] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-27: connection to 10.242.62.102:24013 failed (Connection refused)
>[2013-01-22 13:12:21.993477] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-6: connection to 10.242.62.102:24010 failed (Connection refused)
>[2013-01-22 13:12:21.997278] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-62: connection to 10.242.62.102:24018 failed (Connection refused)
>[2013-01-22 13:12:22.001922] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-20: connection to 10.242.62.102:24012 failed (Connection refused)
>[2013-01-22 13:12:22.005867] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-13: connection to 10.242.62.102:24011 failed (Connection refused)
>[2013-01-22 13:12:22.010437] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-48: connection to 10.242.62.102:24016 failed (Connection refused)
>[2013-01-22 13:12:22.014701] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-55: connection to 10.242.62.102:24017 failed (Connection refused)
>[2013-01-22 13:12:22.018475] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-34: connection to 10.242.62.102:24014 failed (Connection refused)
>[2013-01-22 13:12:22.022388] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-111: connection to 10.242.62.102:24025 failed (Connection refused)
>[2013-01-22 13:12:22.026540] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-83: connection to 10.242.62.102:24021 failed (Connection refused)
>[2013-01-22 13:12:22.030421] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-90: connection to 10.242.62.102:24022 failed (Connection refused)
>[2013-01-22 13:12:22.034491] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-97: connection to 10.242.62.102:24023 failed (Connection refused)
>[2013-01-22 13:12:22.038623] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-76: connection to 10.242.62.102:24020 failed (Connection refused)
>[2013-01-22 13:12:22.043113] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-69: connection to 10.242.62.102:24019 failed (Connection refused)
>[2013-01-22 13:12:22.047253] E [socket.c:1715:socket_connect_finish] 0-hsph_gluster-client-104: connection to 10.242.62.102:24024 failed (Connection refused)
>[2013-01-22 13:12:56.946073] E [dht-common.c:1372:dht_lookup] 0-hsph_gluster-dht: Failed to get hashed subvol for /
>[2013-01-22 13:12:56.946189] W [fuse-bridge.c:513:fuse_attr_cbk] 0-glusterfs-fuse: 82: LOOKUP() / => -1 (Invalid argument)
>[2013-01-22 13:13:48.515474] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-01-22 13:14:53.603273] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-01-22 13:15:28.545239] E [dht-common.c:1372:dht_lookup] 0-hsph_gluster-dht: Failed to get hashed subvol for /
>[2013-01-22 13:16:30.530173] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-01-22 13:17:53.781275] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
>[2013-01-22 13:17:59.557723] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-0: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.557817] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-1: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.557842] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-2: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.557866] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-3: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.557890] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-4: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.557914] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-5: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.557938] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-6: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558048] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-8: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558074] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-9: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558096] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-10: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558119] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-11: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558141] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-12: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558163] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-13: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558211] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-15: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558236] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-16: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558294] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-17: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558317] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-18: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558339] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-19: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558339] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-7: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558410] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-20: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558557] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-14: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558604] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-22: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558634] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-23: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558657] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-24: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558679] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-25: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558700] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-26: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558723] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-27: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558770] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-29: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558799] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-30: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558822] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-31: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558843] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-21: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558844] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-32: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558924] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-33: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558947] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-34: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558985] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-28: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.558998] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-36: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559056] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-37: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559078] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-38: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559100] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-39: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559121] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-40: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559157] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-41: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559205] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-43: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559227] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-44: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559235] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-35: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559255] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-45: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559329] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-46: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559352] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-47: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559372] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-48: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559409] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-42: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559421] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-50: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559488] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-51: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559511] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-52: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559531] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-53: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559551] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-54: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559572] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-55: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559619] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-57: remote operation failed: Transport endpoint is not connected
>[2013-01-22 13:17:59.559634] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-hsph_gluster-client-49: remote operation failed: Transport endpoint is not connected

Actions: View

Attachments on bug 902953: 915661