Bug 1323119

Summary:	TIER : Attach tier fails
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Vivek Das <vdas>
Component:	glusterd	Assignee:	Atin Mukherjee <amukherj>
Status:	CLOSED ERRATA	QA Contact:	Byreddy <bsrirama>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	rhgs-3.1	CC:	amukherj, asrivast, kramdoss, rcyriac, rhinduja, rhs-bugs, rhs-smb, rjoseph, rkavunga, sankarshan, sashinde, storage-qa-internal, vbellur
Target Milestone:	---	Keywords:	Regression, ZStream
Target Release:	RHGS 3.1.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.9-2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1323287 (view as bug list)		Environment:
Last Closed:	2016-06-23 05:15:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1311817, 1323287, 1324156

Description Vivek Das 2016-04-01 10:21:16 UTC

Description of problem:

On a Distributed-Disperse volume 2 x (8 + 4), attached tier distributed-Replicate 4 x 2 = 8. Then went on to do some IOs. After that i did a detach tier which was successful. But when i tried to attach tier again, this is failing. Even after gluster volume stop, restart of glusterd and start the volume i face the same issue.


Version-Release number of selected component (if applicable):

glusterfs-3.7.9-1.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1.Create a Distributed-Disperse volume 2 x (8 + 4)
2.Attach tier distributed-Replicate 4 x 2 = 8
3.Run IOs
4.Detach tier | commit
5.Clean the brick attributes related to detach tier
5.Attach tier.

Actual results:

volume attach-tier: failed: Pre Validation failed...
Brick may be containing or be contained by an existing brick

Expected results:

Attaching tier should be successful
Additional info:

Comment 3 Mohammed Rafi KC 2016-04-04 09:06:52 UTC

upstream patch http://review.gluster.org/#/c/13890/

Comment 4 Mohammed Rafi KC 2016-04-04 09:24:01 UTC

RCA:

For validating new bricks, we use a variable "real_path" which will be filled for every brick in local node. This variable real_path will be calculated when we create a new brick, also when we restore the brick during a glusterd restart.

Now with some reason if an handshake happens from peer node because of the mismatch in data, at this time we are not populating the variable , ie it will null.

If real_path becomes null, then creating a new brick will fail. which means we cannot create or add a brick into the cluster.

Comment 5 Atin Mukherjee 2016-04-04 10:46:56 UTC

This is a regression, and hence setting the keyword.

Comment 8 Atin Mukherjee 2016-04-06 07:36:16 UTC

Downstream patch https://code.engineering.redhat.com/gerrit/71498 posted for review.

Comment 10 rjoseph 2016-04-06 07:40:27 UTC

Upstream master patch      : http://review.gluster.org/13890
Upstream release-3.7 patch : http://review.gluster.org/13914
Downstream patch           : https://code.engineering.redhat.com/gerrit/71498

Comment 12 Mohammed Rafi KC 2016-04-26 11:53:19 UTC

Modified reproducible:

1) Create a  volume on a multinode cluster
2) Kill one node.
3) Do a configuration change (volume set commands like turning off write-behind)
4) Tty to add a brick, or create a new volume.

Comment 13 Byreddy 2016-04-27 05:04:22 UTC

Verified this bug using the build "glusterfs-3.7.9-2.el7r" with below steps.

Steps:
======
1. Created two node cluster with one sample Distribute volume using two bricks.
2. Stopped glusterd on node2
3. Changed the write-behind value using volume set option on node1
4. started glusterd on node2
5. Checked handshake has happened using volume get option on node2
6. Expanded the volume by adding the bricks on node1

All the steps mentioned above worked well.


Note: This issue not always reproducible in last nightly build (3.7.9-1) with above steps.


Fix is working fine, changing the status to verified.

Comment 15 errata-xmlrpc 2016-06-23 05:15:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240