Bug 1335082

Summary:	[Tiering]: Detach tier commit is allowed before rebalance is complete
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	krishnaram Karthick <kramdoss>
Component:	tier	Assignee:	Dan Lambright <dlambrig>
Status:	CLOSED ERRATA	QA Contact:	krishnaram Karthick <kramdoss>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	amukherj, asrivast, dlambrig, rcyriac, rhinduja, rhs-bugs, sankarshan, storage-qa-internal
Target Milestone:	---	Keywords:	Regression, ZStream
Target Release:	RHGS 3.1.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.9-5	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1335558 1335792 (view as bug list)		Environment:
Last Closed:	2016-06-23 05:22:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1311817, 1335558, 1335792

Description krishnaram Karthick 2016-05-11 10:16:55 UTC

Description of problem:
Detach tier commit is allowed to succeed before detach tier is complete and as a result files are left behind in hot tier. 

[root@dhcp47-187 glusterfs]# gluster v tier spiderman detach start
volume detach-tier start: success
ID: 06d6e746-a1a7-4c8c-b4bc-52b261d0d0b4
[root@dhcp47-187 glusterfs]# gluster v tier spiderman detach commit
Removing tier can result in data loss. Do you want to Continue? (y/n) y
volume detach-tier commit: success
Check the detached bricks to ensure all files are migrated.
If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. 

Version-Release number of selected component (if applicable):
glusterfs-server-3.7.9-4.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. create a tiered volume
2. write files to hot tier, write enough files so that tier-rebalance operation takes a while to migrate files from hot tier to cold tier
3. Perform detach tier start and immediately perform a detach tier commit

Actual results:
Files are left unmigrated in hot tier and detach tier commit succeeds

Expected results:
detach tier commit should fail until detach tier operation moves to completed state, so no files are left behind in hot tier unmigrated

Additional info:
sosreports shall be attached shortly.

Comment 4 Dan Lambright 2016-05-11 17:40:54 UTC

This appears to be a downstream only problem. I could recreate it downstream but not on 3.8 or upstream.

Comment 6 Dan Lambright 2016-05-12 16:11:49 UTC

The bug appears to have been that we did not return a failure from glusterd back to the CLI. 

It appears comment #4 is incorrect and the problem exists upstream as well. I will get a patch upstream, and then get it downstream.

Comment 7 Dan Lambright 2016-05-12 16:52:32 UTC

After some further investigation, it appears to have been backported incorrectly downstream. I do not know why this is. At any rate the patch number is 74125.

Comment 9 Atin Mukherjee 2016-05-14 04:23:36 UTC

Downstream patch : https://code.engineering.redhat.com/gerrit/#/c/74126/
3.7 patch : http://review.gluster.org/14327

Master and 3.8 release branch doesn't have this issue.

Comment 11 krishnaram Karthick 2016-05-17 13:34:55 UTC

The issue reported in the bug is no more seen in build - glusterfs-3.7.9-5.

when detach tier commit is run while detach tier is in progress, it fails with an error message 

# gluster v tier spiderman detach commit
Removing tier can result in data loss. Do you want to Continue? (y/n) y
volume detach-tier commit: failed: Brick's in Hot tier is not decommissioned yet. Use gluster volume detach-tier <VOLNAME> <start | commit | force> command instead
Tier command failed

This bug will be moved to verified.

Comment 13 errata-xmlrpc 2016-06-23 05:22:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240