Bug 1404989 - Fail add-brick command if replica count changes
Summary: Fail add-brick command if replica count changes
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute
Version: rhgs-3.2
Hardware: All
OS: All
Target Milestone: ---
: RHGS 3.2.0
Assignee: Karthik U S
QA Contact: Karan Sandha
Depends On:
Blocks: 1351528 1351530 1406411
TreeView+ depends on / blocked
Reported: 2016-12-15 10:19 UTC by Karan Sandha
Modified: 2017-03-23 05:57 UTC (History)
9 users (show)

Fixed In Version: glusterfs-3.8.4-11
Doc Type: Bug Fix
Doc Text:
Previously, the 'gluster volume add-brick' command failed on some nodes when a distributed volume was converted into a replicated volume, and the volume was not mounted, and no lookup had been performed. This could result in inconsistent data across gluster nodes. To avoid this situation, the 'gluster volume add-brick' command is no longer allowed when the replica count has increased and there any replica bricks are unavailable.
Clone Of:
: 1406411 (view as bug list)
Last Closed: 2017-03-23 05:57:15 UTC

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Comment 3 Atin Mukherjee 2016-12-15 11:49:09 UTC
[2016-12-15 11:30:02.898694] E [MSGID: 106054] [glusterd utils.c:11666:glusterd_handle_replicate_brick_ops] 0-        management: Failed to set extended attribute trusted.add-brick : Read-only file system [Read-only file system]

Looks like http://review.gluster.org/12451 has introduced this check.

Comment 4 Atin Mukherjee 2016-12-15 15:35:13 UTC
I managed to git bisect the patch http://review.gluster.org/#/c/15802/ which has caused the regression.

@Pranith - Any guess what from this patch made this regression?

Comment 10 Atin Mukherjee 2016-12-20 13:22:14 UTC
upstream mainline patch http://review.gluster.org/16214 posted for review.

Comment 12 Atin Mukherjee 2016-12-26 10:03:13 UTC
Copying the discussion from upstream bug

(In reply to Mohit Agrawal from comment #4)
> Hi,
> It seems it is expected behavior.As per current dht code in first attempt
> layout sets only when all subvolumes 
> are up otherwise it will not set layout and throws error.

At worst case, we'd need to have a validation in GlusterD to block users end up into this situation otherwise GlusterD will end up into an inconsistent state where in one of the nodes the commit will fail where as in the others it will go through and the transaction will not be roll backed due to the limitation of GlusterD's design.

> Below is the case of plain distributed environment when i have killed one
> brick after start volume then mount is failing as per current dht behavior
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> [root@dhcp10-210 ~]# systemctl restart glusterd.service
> [root@dhcp10-210 ~]# gluster v create test
> volume create: test: success: please start the volume to access data
> [root@dhcp10-210 ~]# gluster v start test
> volume start: test: success
> [root@dhcp10-210 ~]# gluster v status
> Status of volume: test
> Gluster process                             TCP Port  RDMA Port  Online  Pid
> -----------------------------------------------------------------------------
> -
> Brick             49152     0          Y      
> 11117
> Brick             49153     0          Y      
> 11136
> Task Status of Volume test
> -----------------------------------------------------------------------------
> -
> There are no active volume tasks
> [root@dhcp10-210 ~]# kill 11136
> [root@dhcp10-210 ~]# mount -t glusterfs /mnt
> Mount failed. Please check the log file for more detail
> [2016-12-26 06:11:14.871167] W [MSGID: 109005]
> [dht-selfheal.c:2102:dht_selfheal_directory] 0-test-dht: Directory selfheal
> failed: 1 subvolumes down.Not fixing. path = /, gfid = 
> [2016-12-26 06:11:14.880232] W [fuse-bridge.c:767:fuse_attr_cbk]
> 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Stale file handle)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> As of now we think it is a corner case , it would be difficult to provide a
> fix unless there is any data loss in this case.
> Regards
> Mohit Agrawal

Comment 15 Atin Mukherjee 2017-01-05 10:59:34 UTC
http://review.gluster.org/#/c/16330 is posted for review

Comment 17 Atin Mukherjee 2017-01-06 09:40:01 UTC
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/94355

Comment 19 Karan Sandha 2017-01-12 17:30:08 UTC
1) Created a 1*2 volume 
2) Ran a FIO Workload on randomwrites  from fuse mount.
3) killed a brick from first server node(1).
4) now added a brick in the cluster making it arbiter vol 1*2+1

[root@dhcp47-141 ~]# gluster volume add-brick blanc replica 3 arbiter 1  dhcp47-144.lab.eng.blr.redhat.com:/bricks/brick0/blanc

volume add-brick: failed: Brick /bricks/brick0/blanc is down, changing replica count needs all the bricks to be up to avoid data loss

Hence marking this bug verified as per the design change.

Comment 21 Karthik U S 2017-03-07 05:03:41 UTC
Reviewed and updated the doc text, hence removing the needinfo flag.

Comment 24 errata-xmlrpc 2017-03-23 05:57:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.