Bug 1038452

+++ This bug was initially created as a clone of Bug #1028995 +++

Description of problem:
-----------------------
After starting remove-brick on a volume, the bricks were brought down. The status of the remove-brick operations is now shown as stopped.

[root@rhs ~]# gluster v status
Status of volume: test_dis
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 10.70.37.147:/rhs/brick1/b1                       N/A     N       N/A
Brick 10.70.37.147:/rhs/brick1/b2                       N/A     N       N/A
NFS Server on localhost                                 2049    Y       7094
 
Task Status of Volume test_dis
------------------------------------------------------------------------------
Task                 : Remove brick        
ID                   : b3b23f85-f5d5-4e48-a673-4c93a02177ad
Removed bricks:     
10.70.37.147:/rhs/brick1/b1
Status               : stopped             


IMO, brick processes going down, while remove-brick is in progress, should result in a failure in the remove-brick operation, and should not cause it to 'stop'. The status should be shown as failed, instead of stopped.

Version-Release number of selected component (if applicable):
glusterfs 3.4.0.35.1u2rhs

How reproducible:
Always

Steps to Reproduce:
1. Create a distribute volume with 2 bricks, start it, mount it and create data at the mount point.
2. Start remove-brick operation on one of the bricks.
3. Kill glusterfsd processes.
4. Check volume status.

Actual results:
The status of the remove-brick operation is shown as stopped.

Expected results:
The status of the remove-brick operation should be shown as failed, not stopped.

Additional info:
sosreports attached.

--- Additional comment from Shruti Sampat on 2013-11-11 18:23:31 IST ---

Find sosreport at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1028995/

--- Additional comment from Dusmant on 2013-11-11 18:51:52 IST ---

When brick processes are killed while remove-brick is going on, the status of the
remove-brick operation is shown as stopped, instead of failed. glusterfs now expects either a commit or stop of this operation, before another task can be started.

This causes RHSC engine to display the task as stopped. But, neither Commit nor Retain in the UI are enabled. So, a user using the Console can neither start a new task, nor commit/stop the previous task.

This is causing RHSC problem...

--- Additional comment from RamaKasturi on 2013-11-21 13:55:20 IST ---

The above happens with rebalance as well.

When brick processes is killed, rebalance status is shown as stopped , instead of failed.