Description of problem: ----------------------- There were 3 RHGS 3.3.1 nodes in the trusted storage pool. It has the server quorum and client quorum on. There are 3 replica 3 volumes across three nodes in this cluser. When one of the node is rebooted, the bricks are marked down with 'gluster volume status' command, even though there are glusterfsd processes running Version-Release number of selected component (if applicable): ------------------------------------------------------------- RHGS 3.3.1 ( glusterfs-3.8.4-54.15.el7rhgs ) How reproducible: ------------------ Always Steps to Reproduce: ------------------- 1. Create a 3 node trusted storage pool ( gluster cluster ) 2. Create replica 3 volumes 3. Enable server & client quorums 4. Reboot one of the node 5. Check for 'gluster volume status' Actual results: --------------- 'gluster volume status' command reports that the bricks are not running, though the corresponding glusterfsd processes are still running. Expected results: ----------------- After reboot,'gluster volume status' command should report that the correct status of bricks, which in this case is 'up'
I have a suspicion around the following messages in glusterd logs <snip> [2018-07-27 15:49:12.961132] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 6715c775-6021-4f21-a669-83bee56e55c5 [2018-07-27 15:49:12.967504] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2018-07-27 15:49:12.972686] I [MSGID: 106005] [glusterd-handler.c:6122:__glusterd_brick_rpc_notify] 0-management: Brick rhsqa-grafton12.lab.eng.blr.redhat.com:/gluster_bricks/data/data has disconnected from glusterd. [2018-07-27 15:49:12.980700] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2018-07-27 15:49:12.986954] I [MSGID: 106005] [glusterd-handler.c:6122:__glusterd_brick_rpc_notify] 0-management: Brick rhsqa-grafton12.lab.eng.blr.redhat.com:/gluster_bricks/engine/engine has disconnected from glusterd. [2018-07-27 15:49:12.993857] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2018-07-27 15:49:13.000230] I [MSGID: 106005] [glusterd-handler.c:6122:__glusterd_brick_rpc_notify] 0-management: Brick rhsqa-grafton12.lab.eng.blr.redhat.com:/gluster_bricks/vmstore/vmstore has disconnected from glusterd. </snip>
Workaround exists for this issue: After reboot of the node, just need to restart gluster service on that node and 'gluster volume status' reports correct status
Created attachment 1471171 [details] glusterd log file from the rebooted node
Sanju - did we try to reproduce this with latest RHGS bits?
I did the following to reproduce the issue. 1. Formed a 3 node cluster running with RHGS-3.4.0 bits 2. Created 3 replica 3 volumes and started them 3. Enabled server quorum for all volumes gluster volume set <volname> cluster.server-quorum-type server 4. Enabled client quorum for all volumes gluster v set <volname> cluster.quorum-type auto 5. Rebooted one of the node 6. Started glusterd on rebooted node 7. gluster v status shows all bricks online. @Sas, I couldn't reproduce this issue with RHGS-3.4.0 bits. I'm in favour of closing this bug. need your inputs here.
I'm closing this. Please feel free to reopen if the issue persists.
I have hit the same issue while upgrading from RHHI-V 1.1 to RHHI-V 1.5 RHHI-V 1.1 - glusterfs-3.8.4-15.8.el7rhgs RHHI-V 1.5 - glusterfs-3.12.2-25.el7rhgs Post upgrade, the RHVH node was rebooted, whrn the node came up, I could issue gluster volume status and noticed that the brick was down, but after investigating the brick process, those were up and running. So re-opening the bug
Created attachment 1497834 [details] glusterd.log Attaching the glusterd.log as the issue is re-surfaced
Sas - To close down the loop, can you please provide us a setup where this can be replicated so what we can take over and start debugging this? We need to prioritize this bug considering the nature of the problem reported.
Is it a temporary situation, that resolves itself after a short period of time?
(In reply to Yaniv Kaul from comment #29) > Is it a temporary situation, that resolves itself after a short period of > time? No. It doesn't It recovers after glusterd restart on that particular HC node