Bug 1258875 - DHT: Once remove brick start failed in between Remove brick commit should not be allowed
DHT: Once remove brick start failed in between Remove brick commit should not...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity unspecified
: ---
: RHGS 3.1.3
Assigned To: Sakshi
krishnaram Karthick
: ZStream
: 1288448 1330484 (view as bug list)
Depends On:
Blocks: 1278325 1299184 1332370 1333237
  Show dependency treegraph
 
Reported: 2015-09-01 08:47 EDT by RajeshReddy
Modified: 2016-07-31 21:22 EDT (History)
10 users (show)

See Also:
Fixed In Version: glusterfs-3.7.9-4
Doc Type: Bug Fix
Doc Text:
Remove-brick commits were allowed even when remove-brick failed, resulting in data loss when bricks were removed and the remove-brick operation failed because of incomplete data migration from decommissioned bricks. This has been corrected so that failure in remove-brick start prevents commit operations and therefore prevents data loss. If brick data is not important, using the force option forces brick removal regardless.
Story Points: ---
Clone Of:
: 1278325 (view as bug list)
Environment:
Last Closed: 2016-06-23 00:54:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description RajeshReddy 2015-09-01 08:47:16 EDT
Document URL: 
=============
DHT: Once remove brick start failed in between Remove brick commit should not be allowed 

Steps:
========
1. Create a distributed volume with three bricks and mount it on client using FUSE
2. From the mount point create lots of directories and one direcotry with 30k files
3. Remove one of the brick form the volume and while re-blance is in progress delete all directories and files from the mount point and due to this remove-brick operation failed 
4.Though remove-brick operation failed remove-commint job is getting succeeded, 

Expected Result:
================
Remove-brick commit should be allowed only when the remove-brick operation job is passed
Comment 2 Sakshi 2016-05-02 11:09:18 EDT
*** Bug 1330484 has been marked as a duplicate of this bug. ***
Comment 5 Atin Mukherjee 2016-05-05 09:37:09 EDT
Mainline upstream : http://review.gluster.org/#/c/12513/
release-3.7 : http://review.gluster.org/#/c/12513/
Downstream patch : https://code.engineering.redhat.com/gerrit/#/c/73466/
Comment 7 krishnaram Karthick 2016-05-20 10:33:37 EDT
Verified the bug in glusterfs-3.7.9-5

commit fails with an error message.

[root@dhcp46-103 ~]# gluster v remove-brick supernova 10.70.47.128:/bricks/brick1/sn 10.70.47.171:/bricks/brick1/sn 10.70.47.187:/bricks/brick1/sn 10.70.46.103:/bricks/brick1/sn status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0            completed        0:27:19
                            10.70.47.187                0        0Bytes             0             0             0            completed        0:29:12
                            10.70.47.171                0        0Bytes             0             0             0            completed        0:29:8
                            10.70.47.128                0        0Bytes             0             3             0               failed        0:7:17
[root@dhcp46-103 ~]# 
[root@dhcp46-103 ~]# gluster v remove-brick supernova 10.70.47.128:/bricks/brick1/sn 10.70.47.171:/bricks/brick1/sn 10.70.47.187:/bricks/brick1/sn 10.70.46.103:/bricks/brick1/sn commit
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit: failed: Staging failed on 10.70.47.128. Error: use 'force' option as migration has failed

 - using force option indeed allowed to remove the brick


[root@dhcp46-103 ~]# gluster v remove-brick supernova 10.70.47.128:/bricks/brick1/sn 10.70.47.171:/bricks/brick1/sn 10.70.47.187:/bricks/brick1/sn 10.70.46.103:/bricks/brick1/sn force
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success

Moving the bug to verified.
Comment 11 errata-xmlrpc 2016-06-23 00:54:58 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240
Comment 12 krishnaram Karthick 2016-06-24 04:17:27 EDT
*** Bug 1288448 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.