Bug 1323119 - TIER : Attach tier fails
Summary: TIER : Attach tier fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: RHGS 3.1.3
Assignee: Atin Mukherjee
QA Contact: Byreddy
URL:
Whiteboard:
Depends On:
Blocks: 1311817 1323287 1324156
TreeView+ depends on / blocked
 
Reported: 2016-04-01 10:21 UTC by Vivek Das
Modified: 2016-09-17 16:44 UTC (History)
13 users (show)

Fixed In Version: glusterfs-3.7.9-2
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1323287 (view as bug list)
Environment:
Last Closed: 2016-06-23 05:15:20 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1240 0 normal SHIPPED_LIVE Red Hat Gluster Storage 3.1 Update 3 2016-06-23 08:51:28 UTC

Description Vivek Das 2016-04-01 10:21:16 UTC
Description of problem:

On a Distributed-Disperse volume 2 x (8 + 4), attached tier distributed-Replicate 4 x 2 = 8. Then went on to do some IOs. After that i did a detach tier which was successful. But when i tried to attach tier again, this is failing. Even after gluster volume stop, restart of glusterd and start the volume i face the same issue.


Version-Release number of selected component (if applicable):

glusterfs-3.7.9-1.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1.Create a Distributed-Disperse volume 2 x (8 + 4)
2.Attach tier distributed-Replicate 4 x 2 = 8
3.Run IOs
4.Detach tier | commit
5.Clean the brick attributes related to detach tier
5.Attach tier.

Actual results:

volume attach-tier: failed: Pre Validation failed...
Brick may be containing or be contained by an existing brick

Expected results:

Attaching tier should be successful
Additional info:

Comment 3 Mohammed Rafi KC 2016-04-04 09:06:52 UTC
upstream patch http://review.gluster.org/#/c/13890/

Comment 4 Mohammed Rafi KC 2016-04-04 09:24:01 UTC
RCA:

For validating new bricks, we use a variable "real_path" which will be filled for every brick in local node. This variable real_path will be calculated when we create a new brick, also when we restore the brick during a glusterd restart.

Now with some reason if an handshake happens from peer node because of the mismatch in data, at this time we are not populating the variable , ie it will null.

If real_path becomes null, then creating a new brick will fail. which means we cannot create or add a brick into the cluster.

Comment 5 Atin Mukherjee 2016-04-04 10:46:56 UTC
This is a regression, and hence setting the keyword.

Comment 8 Atin Mukherjee 2016-04-06 07:36:16 UTC
Downstream patch https://code.engineering.redhat.com/gerrit/71498 posted for review.

Comment 10 rjoseph 2016-04-06 07:40:27 UTC
Upstream master patch      : http://review.gluster.org/13890
Upstream release-3.7 patch : http://review.gluster.org/13914
Downstream patch           : https://code.engineering.redhat.com/gerrit/71498

Comment 12 Mohammed Rafi KC 2016-04-26 11:53:19 UTC
Modified reproducible:

1) Create a  volume on a multinode cluster
2) Kill one node.
3) Do a configuration change (volume set commands like turning off write-behind)
4) Tty to add a brick, or create a new volume.

Comment 13 Byreddy 2016-04-27 05:04:22 UTC
Verified this bug using the build "glusterfs-3.7.9-2.el7r" with below steps.

Steps:
======
1. Created two node cluster with one sample Distribute volume using two bricks.
2. Stopped glusterd on node2
3. Changed the write-behind value using volume set option on node1
4. started glusterd on node2
5. Checked handshake has happened using volume get option on node2
6. Expanded the volume by adding the bricks on node1

All the steps mentioned above worked well.


Note: This issue not always reproducible in last nightly build (3.7.9-1) with above steps.


Fix is working fine, changing the status to verified.

Comment 15 errata-xmlrpc 2016-06-23 05:15:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240


Note You need to log in before you can comment on or make changes to this bug.