Bug 1697866

Summary: Provide a way to detach a failed node
Product: [Community] GlusterFS Reporter: Sanju <srakonde>
Component: glusterdAssignee: Sanju <srakonde>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: mainlineCC: bmekala, bugs, rhs-bugs, rtalur, sankarshan, storage-qa-internal, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1696334 Environment:
Last Closed: 2019-04-11 15:16:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1696334    

Description Sanju 2019-04-09 08:28:17 UTC 
Description of problem:

When a gluster peer node has failed due to hardware issues, it should be possible to detach it.

Currently, the peer detach command fails because the peer hosts one or more bricks.

If delete of the volume that has that brick is attempted then volume delete fails with "Not all peers are up" error.

One way out is to use a replace-brick command and move the brick to some other node.

However, it might not be possible to replace-brick sometimes. 

A trick that worked for us was to use remove-brick to convert the replica 3 volume to replica 2 and then peer detach the node.


May be the peer detach command can show the trick in output. Something on the lines:


"This peer has one or more bricks. If the peer is lost and is not recoverable then you should use either replace-brick or remove-brick procedure to remove all bricks from the peer and attempt the peer detach again"
Comment 1 Worker Ant 2019-04-09 08:39:48 UTC 
REVIEW: https://review.gluster.org/22534 (glusterd: provide a way to detach failed node) posted (#1) for review on master by Sanju Rakonde
Comment 2 Worker Ant 2019-04-11 15:16:12 UTC 
REVIEW: https://review.gluster.org/22534 (glusterd: provide a way to detach failed node) merged (#3) on master by Atin Mukherjee