Hide Forgot
Description of problem: ========================= Consider a use case where we have 6 storage nodes and 2 x 3 distribute-replicate volume with 1 brick on each node. 2 nodes in the cluster crashed. We have to replace the bricks. Following are the steps suggested to replace the bricks. ( http://documentation-devel.engineering.redhat.com/docs/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/sect-User_Guide-Managing_Volumes-Migrating.html ) 1. Add 2 new nodes into the trusted storage pool. 2. Replace the bricks using the command "gluster volume replace-brick VOLNAME BRICK NEW-BRICK commit force" 3. If replace-brick is successful, detach the nodes that was replaced from the trusted storage pool. ( http://documentation-devel.engineering.redhat.com/docs/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/ch07s02.html ) We are not able to detach the nodes which are dead from the trusted storage pool even though they are not part of any volume. This is because "peer detach" fails when any of the peer in trusted storage pool is in "Disconnected" state. Version-Release number of selected component (if applicable): =============================================================== glusterfs 3.4.0.36rhs built on Oct 22 2013 10:56:18 How reproducible: ==================== Often Steps to Reproduce: ===================== 1. Add 6 servers to the trusted storage pool. 2. stop glusterd on a nodes. 3. execute "peer detach" on any of the node Actual results: ================== peer detach: failed: One of the peers is probably down. Check with 'peer status' Expected results: =================== Detaching the peer should be handled gracefully.
Thank you for submitting this issue for consideration in Red Hat Gluster Storage. The release for which you requested us to review, is now End of Life. Please See https://access.redhat.com/support/policy/updates/rhs/ If you can reproduce this bug against a currently maintained version of Red Hat Gluster Storage, please feel free to file a new report against the current release.
I have reproduced the same error on glusterfs 3.7 exactly the same issue.
If the dead peer doesn't host any of the volumes (bricks to be precise), you should be able to detach it.
I agree, how but I was getting the same error as above. How I solved the issue was by going into /var/lib/glusterfs/peer/ and removing all files here. Then when I restarted the service the peers where gone. This was the only way as I tried every scenario and I kept getting: "peer detach: failed: One of the peers is probably down. Check with 'peer status'" This was a fresh install on ubuntu 14.04 running within my test environment within VMware ESXi 6.0u2 the gluster peer detach <HOSTNAME OR IP> has never worked. The only action I took before trying to detach was installing glusterfs from ppa and peer prob. Anyways found my own work around. Thank you for your reply.