+++ This bug was initially created as a clone of Bug #1027699 +++ Description of problem: ----------------------- In a single-node cluster, when remove-brick is in progress, glusterd is killed and then brought back up. Following this, 'gluster volume status' command fails on the node - [root@rhs ~]# gluster v status test_dis Commit failed on localhost. Please check the log file for more details. The following errors are seen in the glusterd logs - [2013-11-07 03:02:59.984190] I [glusterd-handler.c:3498:__glusterd_handle_status_volume] 0-management: Received status volume req for volume test_dis [2013-11-07 03:02:59.984708] E [glusterd-op-sm.c:1973:_add_remove_bricks_to_dict] 0-management: Failed to get brick count [2013-11-07 03:02:59.984737] E [glusterd-op-sm.c:2037:_add_task_to_dict] 0-management: Failed to add remove bricks to dict [2013-11-07 03:02:59.984753] E [glusterd-op-sm.c:2122:glusterd_aggregate_task_status] 0-management: Failed to add task details to dict [2013-11-07 03:02:59.984768] E [glusterd-syncop.c:993:gd_commit_op_phase] 0-management: Commit of operation 'Volume Status' failed on localhost Version-Release number of selected component (if applicable): glusterfs 3.4.0.35.1u2rhs How reproducible: Always Steps to Reproduce: 1. Create a distribute volume with two bricks, start it, fuse mount it and create some data on the mount point. 2. Start remove-brick of one of the bricks. 3. While remove-brick is in progress, kill glusterd and start it again. 4. Check volume status - # gluster volume status Actual results: The command fails with the following message - Commit failed on localhost. Please check the log file for more details. Expected results: The command should not fail. Additional info: --- Additional comment from Shruti Sampat on 2013-11-07 15:48:43 IST --- --- Additional comment from Dusmant on 2013-11-07 15:50:13 IST --- Because of this problem, RHSC does not update the icon and task does not get updated
REVIEW: http://review.gluster.org/6492 (glusterd: Save/restore/sync rebalance dict) posted (#1) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/6492 (glusterd: Save/restore/sync rebalance dict) posted (#2) for review on master by Kaushal M (kaushal)
COMMIT: http://review.gluster.org/6492 committed in master by Vijay Bellur (vbellur) ------ commit f502e28e8b416f80bd9506ac204948681610b305 Author: Kaushal M <kaushal> Date: Tue Dec 10 11:34:06 2013 +0530 glusterd: Save/restore/sync rebalance dict A dictionary was added to store additional information of a rebalance process, like the bricks being removed in case of a rebalance started by remove-brick. This dictionary wasn't being stored/restored or synced during volume sync, leading to errors like a volume status command failing. These issues have been fixed in this patch. The rebalance dict is now stored/restored and also exported/imported during volume sync. Also, this makes sure that the rebalance dict is only create on remove-brick start. This adds a bricks decommissioned status to the information imported/exported during volume sync. Change-Id: I56fed23dc2de80a96648055fe705e9c3ffd55227 BUG: 1040809 Signed-off-by: Kaushal M <kaushal> Reviewed-on: http://review.gluster.org/6492 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Krishnan Parthasarathi <kparthas> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/6524 (glusterd: Save/restore/sync rebalance dict) posted (#1) for review on release-3.5 by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/6565 (glusterd: Save/restore/sync rebalance dict) posted (#1) for review on release-3.5 by Krishnan Parthasarathi (kparthas)
COMMIT: http://review.gluster.org/6565 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit 3ea6954c120c968ec3b16916cf4fc304b9b4517a Author: Krishnan Parthasarathi <kparthas> Date: Mon Dec 23 14:07:53 2013 +0530 glusterd: Save/restore/sync rebalance dict Backport of http://review.gluster.org/6492 A dictionary was added to store additional information of a rebalance process, like the bricks being removed in case of a rebalance started by remove-brick. This dictionary wasn't being stored/restored or synced during volume sync, leading to errors like a volume status command failing. These issues have been fixed in this patch. The rebalance dict is now stored/restored and also exported/imported during volume sync. Also, this makes sure that the rebalance dict is only create on remove-brick start. This adds a bricks decommissioned status to the information imported/exported during volume sync. Change-Id: I56fed23dc2de80a96648055fe705e9c3ffd55227 BUG: 1040809 Signed-off-by: Kaushal M <kaushal> Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/6565 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/6524 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit 52130cb989d32aa302912d2d75e11f4041db2e72 Author: Kaushal M <kaushal> Date: Tue Dec 10 11:34:06 2013 +0530 glusterd: Save/restore/sync rebalance dict A dictionary was added to store additional information of a rebalance process, like the bricks being removed in case of a rebalance started by remove-brick. This dictionary wasn't being stored/restored or synced during volume sync, leading to errors like a volume status command failing. These issues have been fixed in this patch. The rebalance dict is now stored/restored and also exported/imported during volume sync. Also, this makes sure that the rebalance dict is only create on remove-brick start. This adds a bricks decommissioned status to the information imported/exported during volume sync. BUG: 1040809 Change-Id: I46cee3e4e34a1f3266a20aac43368854594f01bc Signed-off-by: Kaushal M <kaushal> Backport-of: http://review.gluster.org/6492 Reviewed-on: http://review.gluster.org/6524 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report. glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user