Description of problem: After restart one of nodes (tested with 3 node Gluster) some bricks don't go online. All nodes have same CentOS Minimal in with same kernel and same version of Gluster and Ganesha. Version-Release number of selected component (if applicable): CentOS uname -r 3.10.0-514.21.2.el7.x86_64 and also tested with: 3.10.0-514.26.1.el7.x86_64 yum list installed | egrep 'gluster|ganesha' centos-release-gluster310.noarch 1.0-1.el7.centos @extras glusterfs.x86_64 3.10.2-1.el7 @centos-gluster310 glusterfs-api.x86_64 3.10.2-1.el7 @centos-gluster310 glusterfs-cli.x86_64 3.10.2-1.el7 @centos-gluster310 glusterfs-client-xlators.x86_64 3.10.2-1.el7 @centos-gluster310 glusterfs-fuse.x86_64 3.10.2-1.el7 @centos-gluster310 glusterfs-ganesha.x86_64 3.10.2-1.el7 @centos-gluster310 glusterfs-libs.x86_64 3.10.2-1.el7 @centos-gluster310 glusterfs-server.x86_64 3.10.2-1.el7 @centos-gluster310 libntirpc.x86_64 1.4.3-1.el7 @centos-gluster310 nfs-ganesha.x86_64 2.4.5-1.el7 @centos-gluster310 nfs-ganesha-gluster.x86_64 2.4.5-1.el7 @centos-gluster310 userspace-rcu.x86_64 0.7.16-3.el7 @centos-gluster310 How reproducible: Configure Gluster + Ganesha (requires shared storage). Create 16 bricks replicated in 3 nodes. Shutdown all nodes start them again or at least restart one of nodes. Give it enough time to go online (tested with 20 minutes). Processes don't exist for bricks that are offline: ps aux | grep glusterfsd This sometime helps but sometime it doesn't: sudo gluster volume start MyVolume force Reboot usually helps but sometime it needs two reboots or even three. Additional info: http://lists.gluster.org/pipermail/gluster-users/2017-June/031654.html Logs: /var/log/glusterfs/glusterd.log [2017-06-30 13:21:51.332054] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.2 (args: /usr/sbinn /glusterd -p /var/run/glusterd.pid --log-level INFO) [2017-06-30 13:21:51.350768] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-06-30 13:21:51.350813] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-06-30 13:21:51.361177] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.10.2/rpc-transport/rdma.so: cannot open shared oo bject file: No such file or directory [2017-06-30 13:21:51.361202] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not fouu nd on this machine [2017-06-30 13:21:51.361250] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-06-30 13:21:51.361261] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-06-30 13:21:51.365946] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory] [2017-06-30 13:21:51.368176] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31000 [2017-06-30 13:21:51.736010] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-06-30 13:21:51.736494] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-06-30 13:21:51.736542] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-06-30 13:21:51.736580] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:51.741576] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:51.741570] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-06-30 13:21:51.745951] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 2efc7699-b7e9-4087-9343-920de93c1f5c Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-06-30 13:21:51.753452] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-06-30 13:21:52.620796] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 661fcdc0-8c76-4115-8783-944 30925f4a4f, host: glunode2, port: 0 [2017-06-30 13:21:52.659878] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 661fcdc0-- 8c76-4115-8783-9430925f4a4f [2017-06-30 13:21:52.659928] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-06-30 13:21:52.665419] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 661fcdc0-8c76-4115-877 83-9430925f4a4f [2017-06-30 13:21:52.665803] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-06-30 13:21:52.665952] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-06-30 13:21:52.666385] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-06-30 13:21:52.667713] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-06-30 13:21:52.675865] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-06-30 13:21:52.675902] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-06-30 13:21:52.675974] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service [2017-06-30 13:21:53.680769] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-06-30 13:21:53.681585] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-06-30 13:21:53.681619] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-06-30 13:21:53.681674] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-06-30 13:21:53.682210] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-06-30 13:21:53.682236] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-06-30 13:21:53.682285] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-06-30 13:21:53.682789] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-06-30 13:21:53.682808] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-06-30 13:21:53.688289] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /bricks/br0/0 [2017-06-30 13:21:53.688324] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br0/0 on port 49152 [2017-06-30 13:21:53.688363] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.689313] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /bricks/br1/0 [2017-06-30 13:21:53.689339] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br1/0 on port 49154 [2017-06-30 13:21:53.689371] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.692376] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.697182] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.699829] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.702135] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.709954] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.712439] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.714684] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.716927] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.726481] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.729145] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.732228] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.736508] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.740824] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.745470] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.750234] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-06-30 13:21:53.750469] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.750654] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.750822] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.750976] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.751179] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.751335] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.751489] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.751642] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.751793] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.751945] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.752128] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.752286] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.752431] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.752577] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.753193] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.753350] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.753498] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-06-30 13:21:53.753708] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000 [2017-06-30 13:21:53.795942] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 86a82191-60f1-4054-9af8-822 1f0b6026bd, host: glunode1, port: 0 [2017-06-30 13:21:53.806145] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.806835] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.807841] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.808694] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.809916] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.810806] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br2/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.811799] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.812667] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br3/0 has disconnected frr om glusterd. [2017-06-30 13:21:53.813638] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.814453] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br4/0 has disconnected frr om glusterd. [2017-06-30 13:21:53.815432] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.816168] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br5/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.816980] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.817711] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br6/0 has disconnected frr om glusterd. [2017-06-30 13:21:53.818585] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.819313] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/var/lib/glusterd/ss_brick hass disconnected from glusterd. [2017-06-30 13:21:53.820208] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.820877] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw0/0 has disconnected fromm glusterd. [2017-06-30 13:21:53.821848] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.822506] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw1/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.823426] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.824044] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw2/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.824906] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.825623] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw3/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.826432] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.827040] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw4/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.827898] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.828594] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw5/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.829360] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.829919] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/bw6/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.830758] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.831289] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/w0/0 has disconnected fromm glusterd. [2017-06-30 13:21:53.831909] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:53.832428] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/w1/0 has disconnected froo m glusterd. [2017-06-30 13:21:53.832621] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 86a82191-60f1-4054-9aa f8-821f0b6026bd [2017-06-30 13:21:53.832686] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 661fcdc0-8cc 76-4115-8783-9430925f4a4f [2017-06-30 13:21:53.841260] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glunode2 (0), ret: 0, op_ret: 0 [2017-06-30 13:21:53.857354] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 86a82191-- 60f1-4054-9af8-821f0b6026bd [2017-06-30 13:21:53.857407] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-06-30 13:21:53.865212] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000 [2017-06-30 13:21:53.885141] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 86a82191-600 f1-4054-9af8-821f0b6026bd [2017-06-30 13:21:53.901856] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glunode1 (0), ret: 0, op_ret: 0 [2017-06-30 13:21:53.925134] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 86a82191-60f1-4054-9aa f8-821f0b6026bd [2017-06-30 13:21:53.925190] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 86a82191-- 60f1-4054-9af8-821f0b6026bd [2017-06-30 13:21:53.925215] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-06-30 13:21:53.941087] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw1/0 on port 49161 [2017-06-30 13:21:53.955177] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49159 [2017-06-30 13:21:53.955595] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw3/0 on port 49163 [2017-06-30 13:21:53.955668] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br3/0 on port 49155 [2017-06-30 13:21:53.955856] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw0/0 on port 49160 [2017-06-30 13:21:53.959556] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br2/0 on port 49153 [2017-06-30 13:21:53.959648] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br5/0 on port 49157 [2017-06-30 13:21:53.961913] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw5/0 on port 49165 [2017-06-30 13:21:53.964184] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/w1/0 on port 49168 [2017-06-30 13:21:53.965933] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw2/0 on port 49162 [2017-06-30 13:21:53.966980] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/w0/0 on port 49167 [2017-06-30 13:21:53.969164] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 661fcdc0-8c76-4115-877 83-9430925f4a4f [2017-06-30 13:21:53.969218] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 661fcdc0-- 8c76-4115-8783-9430925f4a4f [2017-06-30 13:21:53.969248] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-06-30 13:21:53.974543] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw4/0 on port 49164 [2017-06-30 13:21:53.974635] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/bw6/0 on port 49166 [2017-06-30 13:21:53.978561] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br4/0 on port 49156 [2017-06-30 13:21:53.983654] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/br6/0 on port 49158 [2017-06-30 13:21:57.337446] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:57.338126] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:21:57.338897] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:21:57.339553] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:00.338189] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:00.338833] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:00.340055] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:00.341068] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:03.339040] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:03.340044] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:03.341209] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:03.343133] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:06.339651] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:06.340631] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:06.341795] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:06.342727] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:09.340540] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:09.341545] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:09.342734] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:09.343708] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:12.340941] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:12.341949] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:12.343160] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:12.344062] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:15.341316] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:15.342294] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:15.343484] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:15.344415] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:16.891358] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0 [2017-06-30 13:22:16.897510] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0 [2017-06-30 13:22:16.902158] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0 [2017-06-30 13:22:16.907434] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess 0 [2017-06-30 13:22:16.912148] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc 0 [2017-06-30 13:22:16.916625] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5 [2017-06-30 13:22:16.920967] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt 0 [2017-06-30 13:22:16.925731] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu ster_shared_storage [2017-06-30 13:22:16.929905] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0 [2017-06-30 13:22:16.933676] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1 [2017-06-30 13:22:16.937140] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2 [2017-06-30 13:22:16.940739] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3 [2017-06-30 13:22:16.945151] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4 [2017-06-30 13:22:16.948669] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5 [2017-06-30 13:22:16.952075] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6 [2017-06-30 13:22:16.955502] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0 [2017-06-30 13:22:16.958908] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1 [2017-06-30 13:22:18.343009] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:18.343932] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:18.345087] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:18.346005] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:19.352006] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0 [2017-06-30 13:22:19.357158] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0 [2017-06-30 13:22:19.361059] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0 [2017-06-30 13:22:19.364703] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess 0 [2017-06-30 13:22:19.368392] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc 0 [2017-06-30 13:22:19.372051] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5 [2017-06-30 13:22:19.375698] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt 0 [2017-06-30 13:22:19.379668] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu ster_shared_storage [2017-06-30 13:22:19.383368] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0 [2017-06-30 13:22:19.387033] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1 [2017-06-30 13:22:19.390762] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2 [2017-06-30 13:22:19.394507] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3 [2017-06-30 13:22:19.398248] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4 [2017-06-30 13:22:19.402132] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5 [2017-06-30 13:22:19.405811] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6 [2017-06-30 13:22:19.409233] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0 [2017-06-30 13:22:19.412723] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1 [2017-06-30 13:22:21.342066] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:21.342469] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:21.342903] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:21.343282] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:24.342406] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:24.342775] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:24.343265] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:24.343624] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:24.901200] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0 [2017-06-30 13:22:24.904949] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0 [2017-06-30 13:22:24.908046] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0 [2017-06-30 13:22:24.911147] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess 0 [2017-06-30 13:22:24.914322] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc 0 [2017-06-30 13:22:24.918400] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5 [2017-06-30 13:22:24.921500] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt 0 [2017-06-30 13:22:24.924672] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu ster_shared_storage [2017-06-30 13:22:24.927732] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0 [2017-06-30 13:22:24.930848] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1 [2017-06-30 13:22:24.933937] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2 [2017-06-30 13:22:24.937018] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3 [2017-06-30 13:22:24.940057] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4 [2017-06-30 13:22:24.943080] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5 [2017-06-30 13:22:24.946195] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6 [2017-06-30 13:22:24.949352] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0 [2017-06-30 13:22:24.952493] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1 [2017-06-30 13:22:26.592347] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0 [2017-06-30 13:22:26.595916] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0 [2017-06-30 13:22:26.599018] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0 [2017-06-30 13:22:26.602270] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess 0 [2017-06-30 13:22:26.605392] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc 0 [2017-06-30 13:22:26.608417] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5 [2017-06-30 13:22:26.611447] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt 0 [2017-06-30 13:22:26.616763] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu ster_shared_storage [2017-06-30 13:22:26.619900] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0 [2017-06-30 13:22:26.622953] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1 [2017-06-30 13:22:26.626008] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2 [2017-06-30 13:22:26.629047] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3 [2017-06-30 13:22:26.632073] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4 [2017-06-30 13:22:26.635050] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5 [2017-06-30 13:22:26.638112] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6 [2017-06-30 13:22:26.641208] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0 [2017-06-30 13:22:26.644348] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1 [2017-06-30 13:22:27.342629] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:27.343003] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:27.343466] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:27.343820] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:28.536191] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0 [2017-06-30 13:22:28.539648] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0 [2017-06-30 13:22:28.542787] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0 [2017-06-30 13:22:28.545959] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess 0 [2017-06-30 13:22:28.549139] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc 0 [2017-06-30 13:22:28.552223] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5 [2017-06-30 13:22:28.555324] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt 0 [2017-06-30 13:22:28.561026] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu ster_shared_storage [2017-06-30 13:22:28.564250] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0 [2017-06-30 13:22:28.567494] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1 [2017-06-30 13:22:28.570613] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2 [2017-06-30 13:22:28.573695] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3 [2017-06-30 13:22:28.576803] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4 [2017-06-30 13:22:28.579880] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5 [2017-06-30 13:22:28.583135] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6 [2017-06-30 13:22:28.586267] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0 [2017-06-30 13:22:28.589473] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1 [2017-06-30 13:22:30.342918] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:30.343332] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:30.343797] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:30.344196] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:33.343194] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:33.343569] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:33.344030] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:33.344422] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:35.047264] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0 [2017-06-30 13:22:35.050662] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0 [2017-06-30 13:22:35.053882] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0 [2017-06-30 13:22:35.056927] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess 0 [2017-06-30 13:22:35.059973] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc 0 [2017-06-30 13:22:35.063013] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5 [2017-06-30 13:22:35.068385] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt 0 [2017-06-30 13:22:35.071686] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu ster_shared_storage [2017-06-30 13:22:35.074662] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0 [2017-06-30 13:22:35.077591] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1 [2017-06-30 13:22:35.080491] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2 [2017-06-30 13:22:35.083429] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3 [2017-06-30 13:22:35.086403] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4 [2017-06-30 13:22:35.089346] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5 [2017-06-30 13:22:35.092351] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6 [2017-06-30 13:22:35.095355] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0 [2017-06-30 13:22:35.098513] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1 [2017-06-30 13:22:36.343491] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:36.343865] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:36.344377] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:36.344742] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:37.231988] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ba0 [2017-06-30 13:22:37.235866] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume da0 [2017-06-30 13:22:37.239145] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume de0 [2017-06-30 13:22:37.242241] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dess 0 [2017-06-30 13:22:37.245635] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume docc 0 [2017-06-30 13:22:37.248624] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume br5 [2017-06-30 13:22:37.251697] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume entt 0 [2017-06-30 13:22:37.254662] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluu ster_shared_storage [2017-06-30 13:22:37.257637] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw0 [2017-06-30 13:22:37.260550] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw1 [2017-06-30 13:22:37.263464] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw2 [2017-06-30 13:22:37.266427] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw3 [2017-06-30 13:22:37.269379] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw4 [2017-06-30 13:22:37.272344] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw5 [2017-06-30 13:22:37.275299] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume bw6 [2017-06-30 13:22:37.278302] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w0 [2017-06-30 13:22:37.281562] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume w1 [2017-06-30 13:22:39.343673] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:39.344063] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:39.344557] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:39.344921] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. [2017-06-30 13:22:42.343976] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:42.344408] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br0/0 has disconnected froo m glusterd. [2017-06-30 13:22:42.344867] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-06-30 13:22:42.345267] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glunode0:/bricks/br1/0 has disconnected froo m glusterd. /var/log/glusterfs/bricks/bricks-br0-0.log [2017-06-30 13:17:30.883933] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.10.2 (args: /usr// sbin/glusterfsd -s glunode0 --volfile-id ba0.glunode0.bricks-br0-0 -p /var/lib/glusterd/vols/br0/run/glunode0-bricks-br0-0.pid -S /var/run/gluster/abe3730771485bb6192099 e1f4eb6ad2b.socket --brick-name /bricks/br0/0 -l /var/log/glusterfs/bricks/bricks-br0-0.log --xlator-option *-posix.glusterd-uuid=2efc7699-b7e9-4087-9343-920de93c1f5c -- -brick-port 49152 --xlator-option ba0-server.listen-port=49152) [2017-06-30 13:17:30.901935] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-06-30 13:17:31.083484] I [rpcsvc.c:2237:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2017-06-30 13:17:31.083580] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-br0-server: option 'listen-port' is deprecated, preferred is 'transport.socket.. listen-port', continuing with correction [2017-06-30 13:17:31.119816] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2017-06-30 13:17:31.120930] I [trash.c:2493:init] 0-br0-trash: no option specified for 'eliminate', using NULL [2017-06-30 13:17:31.126937] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-server: option 'rpc-auth.auth-glusterfs' is not recognized [2017-06-30 13:17:31.126981] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-server: option 'rpc-auth.auth-unix' is not recognized [2017-06-30 13:17:31.127008] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-server: option 'rpc-auth.auth-null' is not recognized [2017-06-30 13:17:31.127053] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-server: option 'auth-path' is not recognized [2017-06-30 13:17:31.127068] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-/bricks/br0/0: option 'auth.addr./bricks/br0/0.allow' is not recognized [2017-06-30 13:17:31.127077] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-/bricks/br0/0: option 'auth-path' is not recognized [2017-06-30 13:17:31.127086] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-/bricks/br0/0: option 'auth.login.85fe7a63-8ef0-4220-bdec-59bc4bd05dc6.passworr d' is not recognized [2017-06-30 13:17:31.127095] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-/bricks/br0/0: option 'auth.login./bricks/br0/0.allow' is not recognized [2017-06-30 13:17:31.127123] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-quota: option 'timeout' is not recognized [2017-06-30 13:17:31.127190] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-br0-trash: option 'brick-path' is not recognized Final graph: +------------------------------------------------------------------------------+ 1: volume ba0-posix 2: type storage/posix 3: option glusterd-uuid 2efc7699-b7e9-4087-9343-920de93c1f5c 4: option directory /bricks/br0/0 5: option volume-id 3d6a54f4-6516-46c3-891f-9bbb2666c799 6: end-volume 7: 8: volume ba0-trash 9: type features/trash 10: option trash-dir .trashcan 11: option brick-path /bricks/br0/0 12: option trash-internal-op off 13: subvolumes ba0-posix 14: end-volume 15: 16: volume ba0-changetimerecorder 17: type features/changetimerecorder 18: option db-type sqlite3 19: option hot-brick off 20: option db-name 0.db 21: option db-path /bricks/br0/0/.glusterfs/ 22: option record-exit off 23: option ctr_link_consistency off 24: option ctr_lookupheal_link_timeout 300 25: option ctr_lookupheal_inode_timeout 300 26: option record-entry on 27: option ctr-enabled off 28: option record-counters off 29: option ctr-record-metadata-heat off 30: option sql-db-cachesize 12500 31: option sql-db-wal-autocheckpoint 25000 32: subvolumes ba0-trash 33: end-volume 34: 35: volume ba0-changelog 36: type features/changelog 37: option changelog-brick /bricks/br0/0 38: option changelog-dir /bricks/br0/0/.glusterfs/changelogs 39: option changelog-barrier-timeout 120 40: subvolumes ba0-changetimerecorder 41: end-volume 42: 43: volume ba0-bitrot-stub 44: type features/bitrot-stub 45: option export /bricks/br0/0 46: subvolumes ba0-changelog 47: end-volume 48: 49: volume ba0-access-control 50: type features/access-control 51: subvolumes ba0-bitrot-stub 52: end-volume 53: 54: volume ba0-locks 55: type features/locks 56: subvolumes ba0-access-control 57: end-volume 58: 59: volume ba0-worm 60: type features/worm 61: option worm off 62: option worm-file-level off 63: subvolumes ba0-locks 64: end-volume 65: 66: volume ba0-read-only 67: type features/read-only 68: option read-only off 69: subvolumes ba0-worm 70: end-volume 71: 72: volume ba0-leases 73: type features/leases 74: option leases off 75: subvolumes ba0-read-only 76: end-volume 77: 78: volume ba0-upcall 79: type features/upcall 80: option cache-invalidation on 81: subvolumes ba0-leases 82: end-volume 83: 84: volume ba0-io-threads 85: type performance/io-threads 86: subvolumes ba0-upcall 87: end-volume 88: 89: volume ba0-marker 90: type features/marker 91: option volume-uuid 3d6a54f4-6516-46c3-891f-9bbb2666c799 92: option timestamp-file /var/lib/glusterd/vols/br0/marker.tstamp 93: option quota-version 0 94: option xtime off 95: option gsync-force-xtime off 96: option quota off 97: option inode-quota off 98: subvolumes ba0-io-threads 99: end-volume 100: 101: volume ba0-barrier 102: type features/barrier 103: option barrier disable 104: option barrier-timeout 120 105: subvolumes ba0-marker 106: end-volume 107: 108: volume ba0-index 109: type features/index 110: option index-base /bricks/br0/0/.glusterfs/indices 111: option xattrop-dirty-watchlist trusted.afr.dirty 112: option xattrop-pending-watchlist trusted.afr.ba0- 113: subvolumes ba0-barrier 114: end-volume 115: 116: volume ba0-quota 117: type features/quota 118: option volume-uuid ba0 119: option server-quota off 120: option timeout 0 121: option deem-statfs off 122: subvolumes ba0-index 123: end-volume 124: 125: volume ba0-io-stats 126: type debug/io-stats 127: option unique-id /bricks/br0/0 128: option log-level INFO 129: option latency-measurement off 130: option count-fop-hits off 131: subvolumes ba0-quota 132: end-volume 133: 134: volume /bricks/br0/0 135: type performance/decompounder 136: option auth.addr./bricks/br0/0.allow * 137: option auth-path /bricks/br0/0 138: option auth.login.85fe7a63-8ef0-4220-bdec-59bc4bd05dc6.password c9f3606f-6836-4aa0-a7c6-d61c6f766230 139: option auth.login./bricks/br0/0.allow 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6 140: subvolumes ba0-io-stats 141: end-volume 142: 143: volume ba0-server 144: type protocol/server 145: option transport.socket.listen-port 49152 146: option rpc-auth.auth-glusterfs on 147: option rpc-auth.auth-unix on 148: option rpc-auth.auth-null on 149: option rpc-auth-allow-insecure on 150: option transport-type tcp 151: option transport.address-family inet 152: option auth.login./bricks/br0/0.allow 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6 153: option auth.login.85fe7a63-8ef0-4220-bdec-59bc4bd05dc6.password c9f3606f-6836-4aa0-a7c6-d61c6f766230 154: option auth-path /bricks/br0/0 155: option auth.addr./bricks/br0/0.allow * 156: subvolumes /bricks/br0/0 157: end-volume 158: +------------------------------------------------------------------------------+ [2017-06-30 13:17:31.238188] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.99" [2017-06-30 13:17:31.238219] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6 [2017-06-30 13:17:31.238236] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode1.localdomain-2296-2017/06/300 -12:48:28:451075-br0-client-0-0-1 (version: 3.10.2) [2017-06-30 13:17:31.460308] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.101" [2017-06-30 13:17:31.460336] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6 [2017-06-30 13:17:31.460355] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode2.localdomain-2338-2017/06/300 -13:04:13:178852-br0-client-0-0-1 (version: 3.10.2) [2017-06-30 13:17:32.903615] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.99" [2017-06-30 13:17:32.903653] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6 [2017-06-30 13:17:32.903672] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode1.localdomain-1420-2017/06/300 -12:48:05:50394-br0-client-0-0-1 (version: 3.10.2) [2017-06-30 13:17:33.512188] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.101" [2017-06-30 13:17:33.512211] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6 [2017-06-30 13:17:33.512224] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode2.localdomain-1416-2017/06/300 -13:03:49:963244-br0-client-0-0-1 (version: 3.10.2) [2017-06-30 13:17:33.939453] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.97" [2017-06-30 13:17:33.939481] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6 [2017-06-30 13:17:33.939533] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode0.localdomain-1413-2017/06/300 -13:17:29:866138-br0-client-0-0-0 (version: 3.10.2) [2017-06-30 13:17:57.216018] I [addr.c:182:gf_auth] 0-/bricks/br0/0: allowed = "*", received addr = "10.11.122.97" [2017-06-30 13:17:57.216050] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 85fe7a63-8ef0-4220-bdec-59bc4bd05dc6 [2017-06-30 13:17:57.216073] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-br0-server: accepted client from glunode0.localdomain-2315-2017/06/300 -13:17:53:179027-br0-client-0-0-0 (version: 3.10.2) [2017-06-30 13:18:54.014480] W [socket.c:593:__socket_rwv] 0-glusterfs: readv on 10.11.122.97:24007 failed (No data available) [2017-06-30 13:18:54.014508] E [glusterfsd-mgmt.c:2114:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: glunode0 (No data available) [2017-06-30 13:18:54.014518] I [glusterfsd-mgmt.c:2133:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers [2017-06-30 13:19:04.006807] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-br0-server: disconnecting connection from glunode0.localdomain-2315-2017/06/300 -13:17:53:179027-br0-client-0-0-0 [2017-06-30 13:19:04.007115] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-br0-server: Shutting down connection glunode0.localdomain-2315-2017/06/30-13:11 7:53:179027-br0-client-0-0-0 [2017-06-30 13:19:05.568036] E [socket.c:3203:socket_connect] 0-glusterfs: connection attempt on 10.11.122.97:24007 failed, (Network is unreachable) [2017-06-30 13:19:05.673545] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f216d92fdc5] -->/usr/sbin/glusterfsd(glusterfs_sigwaa iter+0xe5) [0x7f216efc8fd5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x7f216efc8dfb] ) 0-: received signum (15), shutting down [2017-06-30 13:19:06.383751] E [socket.c:3203:socket_connect] 0-glusterfs: connection attempt on 10.11.122.97:24007 failed, (Network is unreachable) [2017-06-30 13:19:06.383788] I [socket.c:3507:socket_submit_request] 0-glusterfs: not connected (priv->connected = 0) [2017-06-30 13:19:06.383799] W [rpc-clnt.c:1693:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (XID: 0x3 Program: Gluster Portmap, ProgVers: 1, Procc : 5) to rpc-transport (glusterfs)
Tested with Gluster version 3.10.3. Issue remains.
Please attach the /etc/ganesha/nfs-ganesha.conf file. Otherwise I don't see anything having to do with ganesha here. ganesha doesn't do anything with bricks. If the bricks don't start that's a glusterd or glusterfsd problem.
Hi Kaleb, it is Gluster issue. Definitelly. I listed Ganesha as a component since I founded this issue on a Gluster cluster with enabled Ganesha and I wanted to give you maximum information about my setup.
This bug reported is against a version of Gluster that is no longer maintained (or has been EOL'd). See https://www.gluster.org/release-schedule/ for the versions currently maintained. As a result this bug is being closed. If the bug persists on a maintained version of gluster or against the mainline gluster repository, request that it be reopened and the Version field be marked appropriately.