Description of problem: When a gluster peer node has failed due to hardware issues, it should be possible to detach it. Currently, the peer detach command fails because the peer hosts one or more bricks. If delete of the volume that has that brick is attempted then volume delete fails with "Not all peers are up" error. One way out is to use a replace-brick command and move the brick to some other node. However, it might not be possible to replace-brick sometimes. A trick that worked for us was to use remove-brick to convert the replica 3 volume to replica 2 and then peer detach the node. May be the peer detach command can show the trick in output. Something on the lines: "This peer has one or more bricks. If the peer is lost and is not recoverable then you should use either replace-brick or remove-brick procedure to remove all bricks from the peer and attempt the peer detach again"
upstream patch: https://review.gluster.org/#/c/glusterfs/+/22534/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:3249