Bug 1560877

Hi!

I have a Distributed-Replica volume with glusterfs, it has 32 bricks distributed in 16 machines (2 bricks per machine).  Almost every day, random bricks get offline and I have to go to the machine that has the brick offline and restart the glusterfs-server. After restart the service, the brick affected go back online.

At the moment that one brick gets offline I see in the affected machines these logs:

glustershd.log:
W [socket.c:590:__socket_rwv] 0-volume1-client-27: readv on 10.0.10.58:49155 failed (No data available)
I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-volume1-client-27: disconnected from volume1-client-27. Client process will keep trying to connect to glusterd until brick's port is available
I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-volume1-client-27: changing port to 49155 (from 0)
E [socket.c:2309:socket_connect_finish] 0-volume1-client-27: connection to 10.0.10.58:49155 failed (Connection refused)


storage-pool.log -> same errors as glusterhd.log

etc-glusterfs-glusterd.vol.log:
W [socket.c:590:__socket_rwv] 0-management: readv on /var/run/gluster/17f9df49125385b6fcafaf23966f47d3.socket failed (No data available)
I [MSGID: 106005] [glusterd-handler.c:5055:__glusterd_brick_rpc_notify] 0-management: Brick ubuntu15:/work/work2/gluster-storage has disconnected from glusterd.


work-work2-gluster-storage.log:

pending frames:
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2018-03-27 05:04:23
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.15
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x7e)[0x7f6af107705e]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x324)[0x7f6af1080784]
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0)[0x7f6af046c4b0]
/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/features/upcall.so(+0xe9da)[0x7f6ae8fab9da]
/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/features/upcall.so(+0xeaac)[0x7f6ae8fabaac]
/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/features/upcall.so(+0x1759)[0x7f6ae8f9e759]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x2f349)[0x7f6af1084349]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3042a)[0x7f6af108542a]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x34)[0x7f6af10856e4]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_wipe+0x12)[0x7f6af1074eb2]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(args_wipe+0x12)[0x7f6af10e1ed2]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(call_stub_destroy+0x39)[0x7f6af1097a19]
/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/performance/io-threads.so(+0x44e7)[0x7f6ae8d964e7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f6af08086ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6af053e41d]
---------


Version-Release number of selected component (if applicable):
glusterfs 3.8.15