Description of problem: The documentation does not explain how to retire bricks/nodes. This is a multi-step process that involves moving data off the brick(s), removing the brick(s) from the volume, then optionally removing the node(s) from the cluster. Here's the proper way to take care of the bricks: gluster volume <volname> remove-brick node1:/brick1 node2:/brick2 start gluster volume <volname> remove-brick node1:/brick1 node2:/brick2 status . . . gluster volume <volname> remove-brick node1:/brick1 node2:/brick2 status <when status shows completed, do the following> gluster volume <volname> remove-brick node1:/brick1 node2:/brick2 commit <answer "y" to the confirmation prompt> Version-Release number of selected component (if applicable): 3.3.0
When I did this procedure before, I did not test to see whether the migrated data was accessible. Today I tried a new test. This is probably going to require a new bug on gluster itself rather than the documentation, but I wanted to get the info written down while it's fresh. I loaded a small 4x2 volume two thirds full and tried to gracefully remove the last set of bricks with the procedure I have outlined above. All of the remaining bricks ran out of disk space during the migration, and there were thousands of migration failures in the log. I started the remove-brick back up, and it again ran out of disk space. A third time completed without migration errors in the log. At this point, I had not issued the commit. After this, I tried to access files in the volume from a client mount. Everything that originally existed on the removed bricks was inaccessible. Final status: Once I did the remove-brick commit, everything magically started working. I'm glad that there was no actual data loss, but if I am removing a set of 4TB bricks that's 75% or so full, it's going to take a really long time for 3TB of data (millions of files) to get migrated. The files that get migrated first will be unavailable for the entire time of the migration effort, which is unacceptable by any standard.