Bug 1467050

Summary: Gluster bricks don't go online after reboot
Product: [Community] GlusterFS Reporter: Jan <jan.h.zak>
Component: coreAssignee: bugs <bugs>
Status: CLOSED EOL QA Contact:
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.10CC: bugs, jan.h.zak
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-20 18:29:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan 2017-07-02 09:34:19 UTC 
Description of problem:
After restart one of nodes (tested with 3 node Gluster) some bricks don't go online. All nodes have same CentOS Minimal in with same kernel and same version of Gluster and Ganesha.

Version-Release number of selected component (if applicable):
CentOS
uname -r
3.10.0-514.21.2.el7.x86_64
and
also tested with:
3.10.0-514.26.1.el7.x86_64

yum list installed | egrep 'gluster|ganesha'
centos-release-gluster310.noarch     1.0-1.el7.centos         @extras

glusterfs.x86_64                     3.10.2-1.el7
@centos-gluster310
glusterfs-api.x86_64                 3.10.2-1.el7
@centos-gluster310
glusterfs-cli.x86_64                 3.10.2-1.el7
@centos-gluster310
glusterfs-client-xlators.x86_64      3.10.2-1.el7
@centos-gluster310
glusterfs-fuse.x86_64                3.10.2-1.el7
@centos-gluster310
glusterfs-ganesha.x86_64             3.10.2-1.el7
@centos-gluster310
glusterfs-libs.x86_64                3.10.2-1.el7
@centos-gluster310
glusterfs-server.x86_64              3.10.2-1.el7
@centos-gluster310
libntirpc.x86_64                     1.4.3-1.el7
 @centos-gluster310
nfs-ganesha.x86_64                   2.4.5-1.el7
 @centos-gluster310
nfs-ganesha-gluster.x86_64           2.4.5-1.el7
 @centos-gluster310
userspace-rcu.x86_64                 0.7.16-3.el7
@centos-gluster310

How reproducible:
Configure Gluster + Ganesha (requires shared storage).

Create 16 bricks replicated in 3 nodes. Shutdown all nodes start them again or at least restart one of nodes.

Give it enough time to go online (tested with 20 minutes).

Processes don't exist for bricks that are offline:
ps aux | grep glusterfsd

This sometime helps but sometime it doesn't:
sudo gluster volume start MyVolume force

Reboot usually helps but sometime it needs two reboots or even three.

Additional info:
http://lists.gluster.org/pipermail/gluster-users/2017-June/031654.html

Logs:

/var/log/glusterfs/glusterd.log

[2017-06-30 13:21:51.332054] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.2 (args: /usr/sbinn
/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-06-30 13:21:51.350768] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-06-30 13:21:51.350813] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory
[2017-06-30 13:21:51.361177] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.10.2/rpc-transport/rdma.so: cannot open shared oo
bject file: No such file or directory
[2017-06-30 13:21:51.361202] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not fouu
nd on this machine
[2017-06-30 13:21:51.361250] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-06-30 13:21:51.361261] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-06-30 13:21:51.365946] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No  
such file or directory]
[2017-06-30 13:21:51.368176] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31000
[2017-06-30 13:21:51.736010] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-06-30 13:21:51.736494] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-06-30 13:21:51.736542] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-06-30 13:21:51.736580] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:51.741576] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:51.741570] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-06-30 13:21:51.745951] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 2efc7699-b7e9-4087-9343-920de93c1f5c
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.socket.listen-backlog 128
  8:     option event-threads 1
  9:     option ping-timeout 0
 10:     option transport.socket.read-fail-log off
 11:     option transport.socket.keepalive-interval 2
 12:     option transport.socket.keepalive-time 10
 13:     option transport-type rdma
 14:     option working-directory /var/lib/glusterd
 15: end-volume
 16:
+------------------------------------------------------------------------------+
[2017-06-30 13:21:51.753452] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-06-30 13:21:52.620796] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 661fcdc0-8c76-4115-8783-944
30925f4a4f, host: glunode2, port: 0
[2017-06-30 13:21:52.659878] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 661fcdc0--
8c76-4115-8783-9430925f4a4f
[2017-06-30 13:21:52.659928] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-06-30 13:21:52.665419] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 661fcdc0-8c76-4115-877
83-9430925f4a4f
[2017-06-30 13:21:52.665803] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-06-30 13:21:52.665952] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-06-30 13:21:52.666385] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-06-30 13:21:52.667713] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-06-30 13:21:52.675865] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-06-30 13:21:52.675902] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-06-30 13:21:52.675974] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-06-30 13:21:53.680769] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-06-30 13:21:53.681585] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-06-30 13:21:53.681619] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-06-30 13:21:53.681674] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-06-30 13:21:53.682210] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-06-30 13:21:53.682236] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-06-30 13:21:53.682285] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-06-30 13:21:53.682789] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-06-30 13:21:53.682808] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-06-30 13:21:53.688289] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /bricks/br0/0
[2017-06-30 13:21:53.688324] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br0/0 on port 49152
[2017-06-30 13:21:53.688363] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.689313] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /bricks/br1/0
[2017-06-30 13:21:53.689339] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br1/0 on port 49154
[2017-06-30 13:21:53.689371] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.692376] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.697182] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.699829] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.702135] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.709954] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.712439] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.714684] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.716927] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.726481] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.729145] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.732228] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.736508] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.740824] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.745470] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.750234] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-06-30 13:21:53.750469] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.750654] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.750822] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.750976] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.751179] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.751335] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.751489] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.751642] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.751793] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.751945] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.752128] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.752286] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.752431] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.752577] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.753193] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.753350] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.753498] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-06-30 13:21:53.753708] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-06-30 13:21:53.795942] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 86a82191-60f1-4054-9af8-822
1f0b6026bd, host: glunode1, port: 0
[2017-06-30 13:21:53.806145] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.806835] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.807841] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.808694] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.809916] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.810806] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br2/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.811799] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.812667] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br3/0 has disconnected frr
om glusterd.
[2017-06-30 13:21:53.813638] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.814453] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br4/0 has disconnected frr
om glusterd.
[2017-06-30 13:21:53.815432] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.816168] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br5/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.816980] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.817711] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br6/0 has disconnected frr
om glusterd.
[2017-06-30 13:21:53.818585] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.819313] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/var/lib/glusterd/ss_brick hass
 disconnected from glusterd.
[2017-06-30 13:21:53.820208] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.820877] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw0/0 has disconnected fromm
 glusterd.
[2017-06-30 13:21:53.821848] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.822506] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw1/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.823426] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.824044] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw2/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.824906] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.825623] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw3/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.826432] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.827040] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw4/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.827898] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.828594] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw5/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.829360] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.829919] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw6/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.830758] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.831289] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/w0/0 has disconnected fromm
 glusterd.
[2017-06-30 13:21:53.831909] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:53.832428] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/w1/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:53.832621] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 86a82191-60f1-4054-9aa
f8-821f0b6026bd
[2017-06-30 13:21:53.832686] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 661fcdc0-8cc
76-4115-8783-9430925f4a4f
[2017-06-30 13:21:53.841260] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glunode2 (0), ret: 0, op_ret: 0
[2017-06-30 13:21:53.857354] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 86a82191--
60f1-4054-9af8-821f0b6026bd
[2017-06-30 13:21:53.857407] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-06-30 13:21:53.865212] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-06-30 13:21:53.885141] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 86a82191-600
f1-4054-9af8-821f0b6026bd
[2017-06-30 13:21:53.901856] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glunode1 (0), ret: 0, op_ret: 0
[2017-06-30 13:21:53.925134] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 86a82191-60f1-4054-9aa
f8-821f0b6026bd
[2017-06-30 13:21:53.925190] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 86a82191--
60f1-4054-9af8-821f0b6026bd
[2017-06-30 13:21:53.925215] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-06-30 13:21:53.941087] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw1/0 on port 49161
[2017-06-30 13:21:53.955177] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49159
[2017-06-30 13:21:53.955595] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw3/0 on port 49163
[2017-06-30 13:21:53.955668] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br3/0 on port 49155
[2017-06-30 13:21:53.955856] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw0/0 on port 49160
[2017-06-30 13:21:53.959556] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br2/0 on port 49153
[2017-06-30 13:21:53.959648] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br5/0 on port 49157
[2017-06-30 13:21:53.961913] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw5/0 on port 49165
[2017-06-30 13:21:53.964184] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/w1/0 on port 49168
[2017-06-30 13:21:53.965933] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw2/0 on port 49162
[2017-06-30 13:21:53.966980] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/w0/0 on port 49167
[2017-06-30 13:21:53.969164] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 661fcdc0-8c76-4115-877
83-9430925f4a4f
[2017-06-30 13:21:53.969218] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 661fcdc0--
8c76-4115-8783-9430925f4a4f
[2017-06-30 13:21:53.969248] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-06-30 13:21:53.974543] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw4/0 on port 49164
[2017-06-30 13:21:53.974635] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw6/0 on port 49166
[2017-06-30 13:21:53.978561] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br4/0 on port 49156
[2017-06-30 13:21:53.983654] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br6/0 on port 49158
[2017-06-30 13:21:57.337446] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:57.338126] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:21:57.338897] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:21:57.339553] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:00.338189] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:00.338833] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:00.340055] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:00.341068] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:03.339040] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:03.340044] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:03.341209] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:03.343133] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:06.339651] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:06.340631] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:06.341795] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:06.342727] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:09.340540] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:09.341545] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:09.342734] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:09.343708] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:12.340941] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:12.341949] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:12.343160] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:12.344062] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:15.341316] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:15.342294] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:15.343484] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:15.344415] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:16.891358] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0
[2017-06-30 13:22:16.897510] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0
[2017-06-30 13:22:16.902158] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0
[2017-06-30 13:22:16.907434] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess
0
[2017-06-30 13:22:16.912148] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc
0
[2017-06-30 13:22:16.916625] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5
[2017-06-30 13:22:16.920967] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt
0
[2017-06-30 13:22:16.925731] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu
ster_shared_storage
[2017-06-30 13:22:16.929905] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0
[2017-06-30 13:22:16.933676] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1
[2017-06-30 13:22:16.937140] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2
[2017-06-30 13:22:16.940739] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3
[2017-06-30 13:22:16.945151] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4
[2017-06-30 13:22:16.948669] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5
[2017-06-30 13:22:16.952075] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6
[2017-06-30 13:22:16.955502] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0
[2017-06-30 13:22:16.958908] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1
[2017-06-30 13:22:18.343009] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:18.343932] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:18.345087] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:18.346005] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:19.352006] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0
[2017-06-30 13:22:19.357158] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0
[2017-06-30 13:22:19.361059] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0
[2017-06-30 13:22:19.364703] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess
0
[2017-06-30 13:22:19.368392] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc
0
[2017-06-30 13:22:19.372051] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5
[2017-06-30 13:22:19.375698] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt
0
[2017-06-30 13:22:19.379668] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu
ster_shared_storage
[2017-06-30 13:22:19.383368] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0
[2017-06-30 13:22:19.387033] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1
[2017-06-30 13:22:19.390762] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2
[2017-06-30 13:22:19.394507] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3
[2017-06-30 13:22:19.398248] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4
[2017-06-30 13:22:19.402132] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5
[2017-06-30 13:22:19.405811] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6
[2017-06-30 13:22:19.409233] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0
[2017-06-30 13:22:19.412723] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1
[2017-06-30 13:22:21.342066] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:21.342469] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:21.342903] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:21.343282] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:24.342406] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:24.342775] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:24.343265] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:24.343624] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:24.901200] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0
[2017-06-30 13:22:24.904949] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0
[2017-06-30 13:22:24.908046] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0
[2017-06-30 13:22:24.911147] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess
0
[2017-06-30 13:22:24.914322] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc
0
[2017-06-30 13:22:24.918400] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5
[2017-06-30 13:22:24.921500] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt
0
[2017-06-30 13:22:24.924672] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu
ster_shared_storage
[2017-06-30 13:22:24.927732] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0
[2017-06-30 13:22:24.930848] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1
[2017-06-30 13:22:24.933937] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2
[2017-06-30 13:22:24.937018] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3
[2017-06-30 13:22:24.940057] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4
[2017-06-30 13:22:24.943080] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5
[2017-06-30 13:22:24.946195] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6
[2017-06-30 13:22:24.949352] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0
[2017-06-30 13:22:24.952493] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1
[2017-06-30 13:22:26.592347] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0
[2017-06-30 13:22:26.595916] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0
[2017-06-30 13:22:26.599018] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0
[2017-06-30 13:22:26.602270] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess
0
[2017-06-30 13:22:26.605392] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc
0
[2017-06-30 13:22:26.608417] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5
[2017-06-30 13:22:26.611447] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt
0
[2017-06-30 13:22:26.616763] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu
ster_shared_storage
[2017-06-30 13:22:26.619900] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0
[2017-06-30 13:22:26.622953] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1
[2017-06-30 13:22:26.626008] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2
[2017-06-30 13:22:26.629047] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3
[2017-06-30 13:22:26.632073] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4
[2017-06-30 13:22:26.635050] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5
[2017-06-30 13:22:26.638112] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6
[2017-06-30 13:22:26.641208] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0
[2017-06-30 13:22:26.644348] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1
[2017-06-30 13:22:27.342629] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:27.343003] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:27.343466] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:27.343820] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:28.536191] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0
[2017-06-30 13:22:28.539648] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0
[2017-06-30 13:22:28.542787] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0
[2017-06-30 13:22:28.545959] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess
0
[2017-06-30 13:22:28.549139] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc
0
[2017-06-30 13:22:28.552223] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5
[2017-06-30 13:22:28.555324] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt
0
[2017-06-30 13:22:28.561026] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu
ster_shared_storage
[2017-06-30 13:22:28.564250] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0
[2017-06-30 13:22:28.567494] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1
[2017-06-30 13:22:28.570613] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2
[2017-06-30 13:22:28.573695] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3
[2017-06-30 13:22:28.576803] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4
[2017-06-30 13:22:28.579880] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5
[2017-06-30 13:22:28.583135] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6
[2017-06-30 13:22:28.586267] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0
[2017-06-30 13:22:28.589473] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1
[2017-06-30 13:22:30.342918] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:30.343332] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:30.343797] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:30.344196] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:33.343194] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:33.343569] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:33.344030] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:33.344422] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:35.047264] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0
[2017-06-30 13:22:35.050662] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0
[2017-06-30 13:22:35.053882] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0
[2017-06-30 13:22:35.056927] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess
0
[2017-06-30 13:22:35.059973] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc
0
[2017-06-30 13:22:35.063013] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5
[2017-06-30 13:22:35.068385] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt
0
[2017-06-30 13:22:35.071686] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu
ster_shared_storage
[2017-06-30 13:22:35.074662] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0
[2017-06-30 13:22:35.077591] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1
[2017-06-30 13:22:35.080491] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2
[2017-06-30 13:22:35.083429] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3
[2017-06-30 13:22:35.086403] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4
[2017-06-30 13:22:35.089346] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5
[2017-06-30 13:22:35.092351] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6
[2017-06-30 13:22:35.095355] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0
[2017-06-30 13:22:35.098513] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1
[2017-06-30 13:22:36.343491] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:36.343865] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:36.344377] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:36.344742] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:37.231988] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0
[2017-06-30 13:22:37.235866] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0
[2017-06-30 13:22:37.239145] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0
[2017-06-30 13:22:37.242241] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess
0
[2017-06-30 13:22:37.245635] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc
0
[2017-06-30 13:22:37.248624] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5
[2017-06-30 13:22:37.251697] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt
0
[2017-06-30 13:22:37.254662] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu
ster_shared_storage
[2017-06-30 13:22:37.257637] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0
[2017-06-30 13:22:37.260550] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1
[2017-06-30 13:22:37.263464] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2
[2017-06-30 13:22:37.266427] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3
[2017-06-30 13:22:37.269379] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4
[2017-06-30 13:22:37.272344] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5
[2017-06-30 13:22:37.275299] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6
[2017-06-30 13:22:37.278302] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0
[2017-06-30 13:22:37.281562] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1
[2017-06-30 13:22:39.343673] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:39.344063] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:39.344557] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:39.344921] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:42.343976] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:42.344408] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo
m glusterd.
[2017-06-30 13:22:42.344867] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-06-30 13:22:42.345267] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo
m glusterd.




/var/log/glusterfs/bricks/bricks-br0-0.log

[2017-06-30 13:17:30.883933] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.10.2 (args: /usr//
sbin/glusterfsd -s glunode0 --volfile-id ba0.glunode0.bricks-br0-0 -p /var/lib/glusterd/vols/br0/run/glunode0-bricks-br0-0.pid -S /var/run/gluster/abe3730771485bb6192099
e1f4eb6ad2b.socket --brick-name /bricks/br0/0 -l /var/log/glusterfs/bricks/bricks-br0-0.log --xlator-option *-posix.glusterd-uuid=2efc7699-b7e9-4087-9343-920de93c1f5c --
-brick-port 49152 --xlator-option ba0-server.listen-port=49152)
[2017-06-30 13:17:30.901935] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-06-30 13:17:31.083484] I [rpcsvc.c:2237:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2017-06-30 13:17:31.083580] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-br0-server: option 'listen-port' is deprecated, preferred is 'transport.socket..
listen-port', continuing with correction
[2017-06-30 13:17:31.119816] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled.
[2017-06-30 13:17:31.120930] I [trash.c:2493:init] 0-br0-trash: no option specified for 'eliminate', using NULL
[2017-06-30 13:17:31.126937] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2017-06-30 13:17:31.126981] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-server: option 'rpc-auth.auth-unix' is not recognized
[2017-06-30 13:17:31.127008] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-server: option 'rpc-auth.auth-null' is not recognized
[2017-06-30 13:17:31.127053] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-server: option 'auth-path' is not recognized
[2017-06-30 13:17:31.127068] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-/bricks/br0/0: option 'auth.addr./bricks/br0/0.allow' is not recognized
[2017-06-30 13:17:31.127077] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-/bricks/br0/0: option 'auth-path' is not recognized
[2017-06-30 13:17:31.127086] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-/bricks/br0/0: option 'auth.login.85fe7a63-8ef0-4220-bdec-59bc4bd05dc6.passworr
d' is not recognized
[2017-06-30 13:17:31.127095] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-/bricks/br0/0: option 'auth.login./bricks/br0/0.allow' is not recognized
[2017-06-30 13:17:31.127123] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-quota: option 'timeout' is not recognized
[2017-06-30 13:17:31.127190] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-trash: option 'brick-path' is not recognized
Final graph:
+------------------------------------------------------------------------------+
  1: volume ba0-posix
  2:     type storage/posix
  3:     option glusterd-uuid 2efc7699-b7e9-4087-9343-920de93c1f5c
  4:     option directory /bricks/br0/0
  5:     option volume-id 3d6a54f4-6516-46c3-891f-9bbb2666c799
  6: end-volume
  7:
  8: volume ba0-trash
  9:     type features/trash
 10:     option trash-dir .trashcan
 11:     option brick-path /bricks/br0/0
 12:     option trash-internal-op off
 13:     subvolumes ba0-posix
 14: end-volume
 15:
 16: volume ba0-changetimerecorder
 17:     type features/changetimerecorder
 18:     option db-type sqlite3
 19:     option hot-brick off
 20:     option db-name 0.db
 21:     option db-path /bricks/br0/0/.glusterfs/
 22:     option record-exit off
 23:     option ctr_link_consistency off
 24:     option ctr_lookupheal_link_timeout 300
 25:     option ctr_lookupheal_inode_timeout 300
 26:     option record-entry on
 27:     option ctr-enabled off
 28:     option record-counters off
 29:     option ctr-record-metadata-heat off
 30:     option sql-db-cachesize 12500
 31:     option sql-db-wal-autocheckpoint 25000
 32:     subvolumes ba0-trash
 33: end-volume
 34:
 35: volume ba0-changelog
 36:     type features/changelog
 37:     option changelog-brick /bricks/br0/0
 38:     option changelog-dir /bricks/br0/0/.glusterfs/changelogs
 39:     option changelog-barrier-timeout 120
 40:     subvolumes ba0-changetimerecorder
 41: end-volume
 42:
 43: volume ba0-bitrot-stub
 44:     type features/bitrot-stub
 45:     option export /bricks/br0/0
 46:     subvolumes ba0-changelog
 47: end-volume
 48:
 49: volume ba0-access-control
 50:     type features/access-control
 51:     subvolumes ba0-bitrot-stub
 52: end-volume
 53:
 54: volume ba0-locks
 55:     type features/locks
 56:     subvolumes ba0-access-control
 57: end-volume
 58:
 59: volume ba0-worm
 60:     type features/worm
 61:     option worm off
 62:     option worm-file-level off
 63:     subvolumes ba0-locks
 64: end-volume
 65:
 66: volume ba0-read-only
 67:     type features/read-only
 68:     option read-only off
 69:     subvolumes ba0-worm
 70: end-volume
 71:
 72: volume ba0-leases
 73:     type features/leases
 74:     option leases off
 75:     subvolumes ba0-read-only
 76: end-volume
 77:
 78: volume ba0-upcall
 79:     type features/upcall
 80:     option cache-invalidation on
 81:     subvolumes ba0-leases
 82: end-volume
 83:
 84: volume ba0-io-threads
 85:     type performance/io-threads
 86:     subvolumes ba0-upcall
 87: end-volume
 88:
 89: volume ba0-marker
 90:     type features/marker
 91:     option volume-uuid 3d6a54f4-6516-46c3-891f-9bbb2666c799
 92:     option timestamp-file /var/lib/glusterd/vols/br0/marker.tstamp
 93:     option quota-version 0
 94:     option xtime off
 95:     option gsync-force-xtime off
 96:     option quota off
 97:     option inode-quota off
 98:     subvolumes ba0-io-threads
 99: end-volume
100:
101: volume ba0-barrier
102:     type features/barrier
103:     option barrier disable
104:     option barrier-timeout 120
105:     subvolumes ba0-marker
106: end-volume
107:
108: volume ba0-index
109:     type features/index
110:     option index-base /bricks/br0/0/.glusterfs/indices
111:     option xattrop-dirty-watchlist trusted.afr.dirty
112:     option xattrop-pending-watchlist trusted.afr.ba0-
113:     subvolumes ba0-barrier
114: end-volume
115:
116: volume ba0-quota
117:     type features/quota
118:     option volume-uuid ba0
119:     option server-quota off
120:     option timeout 0
121:     option deem-statfs off
122:     subvolumes ba0-index
123: end-volume
124:
125: volume ba0-io-stats
126:     type debug/io-stats
127:     option unique-id /bricks/br0/0
128:     option log-level INFO
129:     option latency-measurement off
130:     option count-fop-hits off
131:     subvolumes ba0-quota
132: end-volume
133:
134: volume /bricks/br0/0
135:     type performance/decompounder
136:     option auth.addr./bricks/br0/0.allow *
137:     option auth-path /bricks/br0/0
138:     option auth.login.85fe7a63-8ef0-4220-bdec-59bc4bd05dc6.password c9f3606f-6836-4aa0-a7c6-d61c6f766230
139:     option auth.login./bricks/br0/0.allow 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6
140:     subvolumes ba0-io-stats
141: end-volume
142:
143: volume ba0-server
144:     type protocol/server
145:     option transport.socket.listen-port 49152
146:     option rpc-auth.auth-glusterfs on
147:     option rpc-auth.auth-unix on
148:     option rpc-auth.auth-null on
149:     option rpc-auth-allow-insecure on
150:     option transport-type tcp
151:     option transport.address-family inet
152:     option auth.login./bricks/br0/0.allow 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6
153:     option auth.login.85fe7a63-8ef0-4220-bdec-59bc4bd05dc6.password c9f3606f-6836-4aa0-a7c6-d61c6f766230
154:     option auth-path /bricks/br0/0
155:     option auth.addr./bricks/br0/0.allow *
156:     subvolumes /bricks/br0/0
157: end-volume
158:
+------------------------------------------------------------------------------+
[2017-06-30 13:17:31.238188] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.99"
[2017-06-30 13:17:31.238219] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6
[2017-06-30 13:17:31.238236] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode1.localdomain-2296-2017/06/300
-12:48:28:451075-br0-client-0-0-1 (version: 3.10.2)
[2017-06-30 13:17:31.460308] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.101"
[2017-06-30 13:17:31.460336] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6
[2017-06-30 13:17:31.460355] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode2.localdomain-2338-2017/06/300
-13:04:13:178852-br0-client-0-0-1 (version: 3.10.2)
[2017-06-30 13:17:32.903615] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.99"
[2017-06-30 13:17:32.903653] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6
[2017-06-30 13:17:32.903672] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode1.localdomain-1420-2017/06/300
-12:48:05:50394-br0-client-0-0-1 (version: 3.10.2)
[2017-06-30 13:17:33.512188] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.101"
[2017-06-30 13:17:33.512211] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6
[2017-06-30 13:17:33.512224] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode2.localdomain-1416-2017/06/300
-13:03:49:963244-br0-client-0-0-1 (version: 3.10.2)
[2017-06-30 13:17:33.939453] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.97"
[2017-06-30 13:17:33.939481] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6
[2017-06-30 13:17:33.939533] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode0.localdomain-1413-2017/06/300
-13:17:29:866138-br0-client-0-0-0 (version: 3.10.2)
[2017-06-30 13:17:57.216018] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.97"
[2017-06-30 13:17:57.216050] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6
[2017-06-30 13:17:57.216073] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode0.localdomain-2315-2017/06/300
-13:17:53:179027-br0-client-0-0-0 (version: 3.10.2)
[2017-06-30 13:18:54.014480] W [socket.c:593:__socket_rwv] 0-glusterfs: readv on 10.11.122.97:24007 failed (No data available)
[2017-06-30 13:18:54.014508] E [glusterfsd-mgmt.c:2114:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: glunode0 (No data available)
[2017-06-30 13:18:54.014518] I [glusterfsd-mgmt.c:2133:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2017-06-30 13:19:04.006807] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-br0-server: disconnecting connection from glunode0.localdomain-2315-2017/06/300
-13:17:53:179027-br0-client-0-0-0
[2017-06-30 13:19:04.007115] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-br0-server: Shutting down connection glunode0.localdomain-2315-2017/06/30-13:11
7:53:179027-br0-client-0-0-0
[2017-06-30 13:19:05.568036] E [socket.c:3203:socket_connect] 0-glusterfs: connection attempt on 10.11.122.97:24007 failed, (Network is unreachable)
[2017-06-30 13:19:05.673545] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f216d92fdc5] -->/usr/sbin/glusterfsd(glusterfs_sigwaa
iter+0xe5) [0x7f216efc8fd5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x7f216efc8dfb] ) 0-: received signum (15), shutting down
[2017-06-30 13:19:06.383751] E [socket.c:3203:socket_connect] 0-glusterfs: connection attempt on 10.11.122.97:24007 failed, (Network is unreachable)
[2017-06-30 13:19:06.383788] I [socket.c:3507:socket_submit_request] 0-glusterfs: not connected (priv->connected = 0)
[2017-06-30 13:19:06.383799] W [rpc-clnt.c:1693:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (XID: 0x3 Program: Gluster Portmap, ProgVers: 1, Procc
: 5) to rpc-transport (glusterfs)
Comment 1 Jan 2017-07-03 09:12:34 UTC 
Tested with Gluster version 3.10.3. Issue remains.
Comment 2 Kaleb KEITHLEY 2017-07-03 11:29:52 UTC 
Please attach the /etc/ganesha/nfs-ganesha.conf file.

Otherwise I don't see anything having to do with ganesha here. ganesha doesn't do anything with bricks. If the bricks don't start that's a glusterd or glusterfsd problem.
Comment 3 Jan 2017-07-03 18:53:51 UTC 
Hi Kaleb,

it is Gluster issue. Definitelly.

I listed Ganesha as a component since I founded this issue on a Gluster cluster with enabled Ganesha and I wanted to give you maximum information about my setup.
Comment 4 Shyamsundar 2018-06-20 18:29:57 UTC 
This bug reported is against a version of Gluster that is no longer maintained
(or has been EOL'd). See https://www.gluster.org/release-schedule/ for the
versions currently maintained.

As a result this bug is being closed.

If the bug persists on a maintained version of gluster or against the mainline
gluster repository, request that it be reopened and the Version field be marked
appropriately.