Bug 1284824

Summary:	[Doc RFE] Add information regarding Support add/remove bricks for Tiering
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Anjana Suparna Sriram <asriram>
Component:	doc-Administration_Guide	Assignee:	Divya <divya>
doc-Administration_Guide sub component:	Default	QA Contact:	Nag Pavan Chilakam <nchilaka>
Status:	CLOSED CURRENTRELEASE	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	asriram, asrivast, divya, dlambrig, mhideo, mzywusko, nlevinki, rhs-bugs, rwheeler, storage-doc
Version:	rhgs-3.1	Keywords:	Documentation, FutureFeature, ZStream
Target Milestone:	---	Flags:	divya: needinfo- divya: needinfo- divya: needinfo-
Target Release:	RHGS 3.1.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1270801
Bug Blocks:

Description Anjana Suparna Sriram 2015-11-24 09:55:43 UTC

Document URL: Received the following information from QE:

Hi,
Following are the steps and cautions to be taken for adding bricks either to hot or cold tier:
1)Currently there is no easy and non disruptive( wrt to tiering functionality) method to expand either a hot or a cold or both the tiers.
If an user wants to provision more storage by adding bricks of a tiered volume, he/she needs to follow the below mentioned work-around:
1)identify the tier,
2)It is always better to stop IOs for the time being, though not a must
3)Now issue the detach tier command "gluster volume tier <volname> detach start". This command triggers data migration from hot to cold tier
4)Track the detach tier start status using "gluster volume tier <volname> detach status". Once the status shows as completed on all the nodes, then go ahead with the next step
5)Issue a detach tier commit to remove the hot tier completely. use "gluster volume tier <volname> detach commit"
6)Once the detach tier commit passes, you can notice that your volume is no more a tier volume(using volume info)
7)To expand the cold tier:Now add bricks to the regular non-tiered volume. Always add a fresh set of bricks and don't repurpose bricks until unless they are clearly wiped out. Use "gluster volume add-brick <vname>"
8)Now fix the layout of existing bricks and directories by run a "gluster volume rebalance <VOLNAME> fix-layout start"
9)Wait for the fix-layout to complete successfully.
10)If required you may run the rebalance command to move and balance files so as to optimize evenly the available space on all the bricks.
11)Attaching(and for adding bricks to hot tier) Hot Tier:use the command "gluster v tier <vname> attach". Always use a new set of bricks and don't repurpose previously used hotbricks until unless they are clearly wiped

Warnings/Recommendations:
1)it is always recommended to provision you storage liberally and generously before attaching a tier
2)Always add a fresh set of bricks and don't re-purpose bricks until unless they are clearly wiped out.
3)We recommend to stop IOs while performing add bricks procedure for a tiered volume.
4)Always run fix-layout on adding of any cold brick set, as we currently don't allow fix-layout once , hot tier is attached. Without fixing layout, you may end up using your new bricks ineffectively.

Comment 2 Divya 2015-12-15 10:00:25 UTC

I have requested Dan to sign-off on "the Expanding a Tiered Volume section" in bug 1270801. 

Expanding a Tiered Volume - http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20(html-single)/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Red_Hat_Storage_Volumes-Expanding_Tiered_Volume

Will move this bug to ON_QA after I receive sign-off on the doc.

Comment 3 Dan Lambright 2015-12-22 14:56:09 UTC

This section

"⁠10.3.1.2. Expanding a Hot Tier Volume"

misled me, it suggests you can run the "attach" command against an existing tiered volume. I understand the Prerequisite immediately above states that detach is necessary, but it probably should be reiterated in section 10.3.1.2. In my reading it is easy to miss that prerequisite.

If you want to expand the hot tier of a tiered volume, the procedure is

1. detach the tier
2. re-attach the tier with both the old and new (expanded) bricks

Comment 4 Divya 2015-12-23 10:13:35 UTC

(In reply to Dan Lambright from comment #3)
> This section
> 
> "⁠10.3.1.2. Expanding a Hot Tier Volume"
> 
> misled me, it suggests you can run the "attach" command against an existing
> tiered volume. I understand the Prerequisite immediately above states that
> detach is necessary, but it probably should be reiterated in section
> 10.3.1.2. In my reading it is easy to miss that prerequisite.
> 
> If you want to expand the hot tier of a tiered volume, the procedure is
> 
> 1. detach the tier
> 2. re-attach the tier with both the old and new (expanded) bricks

Agree with you Dan. There are chances that user might miss the Prerequisite listed above.

Updated the doc based on your feedback. Link to the latest doc: http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20(html-single)/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Red_Hat_Storage_Volumes-Expanding_Tiered_Volume

Could you please review the changes and sign-off?

Thanks!

Comment 5 Divya 2015-12-28 10:06:23 UTC

I have deleted the prerequisite of detaching a tier in "⁠10.3.1.2. Expanding a Tiered Volume" section and added it as Step 1 in "⁠10.3.1.2. Expanding a Cold Tier Volume" section to ensure that the user do not miss this critical information.

Comment 6 Dan Lambright 2015-12-29 15:51:49 UTC

Looks good

sign-off

Comment 7 Divya 2015-12-30 04:25:14 UTC

Based on Comment 6, moving the bug to ON_QA.

Link to the latest doc: http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20(html-single)/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Red_Hat_Storage_Volumes-Expanding_Tiered_Volume

Comment 8 Nag Pavan Chilakam 2016-01-12 14:25:21 UTC

I don't see a need of mentioning the first step in 
http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20(html-single)/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Data_Tieirng-Detach_Tier


Stop the tier by executing the following command
# gluster volume rebalance stop tier VOLNAME
For example
# gluster volume rebalance stop tier test-volume

Comment 9 Dan Lambright 2016-01-12 14:35:48 UTC

""You must ensure that there are no I/O happening when detaching a tier.""

I believe this warning in the detach tier section should be removed. It is not necessary and would be disruptive to the user.

Comment 10 Divya 2016-01-13 09:56:14 UTC

Based on Comment 8 and 9, I have deleted the following from "11.7.1. Detaching a Tier of a Geo-replicated Volume" and "11.7. Detaching a Tier from a Volume" sections:

----
Stop the tier by executing the following command
# gluster volume rebalance stop tier VOLNAME
For example
# gluster volume rebalance stop tier test-volume
---

I have also deleted "You must ensure that there are no I/O happening when detaching a tier." warning in the "11.7. Detaching a Tier from a Volume" section.

Link to the latest doc: http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20%28html-single%29/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Data_Tieirng-Detach_Tier

Comment 11 Dan Lambright 2016-01-13 11:41:45 UTC

the changes look good.

Comment 12 Nag Pavan Chilakam 2016-01-14 09:16:55 UTC

Following are my comments for 11.7. Detaching a Tier from a Volume section:

Comment#1:

Step#1:gluster volume detach-tier VOLNAME start

Comment:I thought we would be using new cli ie, gluster volume detach VOLNAME tier start "eg:gluster v tier countme detach  start"


Comment#2:
Step#2 
comment: same as above, we should be using "gluster v tier countme detach status"

Comment#3:
==> Same to be implemented for detach tier in Geo-rep session too"



Comment#4:
10.3.1.1. Expanding a Cold Tier Volume--> I feel it is incomplete. We are telling admin to detach tier, add new bricks.
We should also tell him to reattach the tier, using normal tier attach command




Comment#5:
We must be putting a note or somekind of highlighting done , for both expanding and shrinking tier volumes, saying " it is a disruptive opeation"

Comment 13 Dan Lambright 2016-01-14 16:24:50 UTC

Agree with items 1-3 in comment #12, items 4-5 make sense and you can discuss with Alok as well.

Comment 14 Divya 2016-01-18 09:46:23 UTC

(In reply to Dan Lambright from comment #13)
> Agree with items 1-3 in comment #12, items 4-5 make sense and you can
> discuss with Alok as well.

I will incorporate items 1 - 3 listed in Comment 12.

Alok,

Could you please review suggestions/comments 4 and 5 in Comment 12 and share your thoughts.

Comment 15 Alok 2016-01-20 14:19:49 UTC

Will be good to include steps 4 and 5. Please explain briefly about "What happens between detach tier and attach tier" e.g. fix-layout taking time etc, instead of just saying that the operation is disruptive.

Comment 16 Divya 2016-01-22 11:04:39 UTC

(In reply to Alok from comment #15)
> Will be good to include steps 4 and 5. Please explain briefly about "What
> happens between detach tier and attach tier" e.g. fix-layout taking time
> etc, instead of just saying that the operation is disruptive.

Dan,

Could you please provide me the text that needs to be added to explain this disruptive operation!

Comment 17 Dan Lambright 2016-01-22 14:22:27 UTC

"What happens between detach tier and attach tier" e.g. fix-layout taking time
> etc, instead of just saying that the operation is disruptive.

I believe Alok is referring to the internal "fix-layout" operation here. 

Check with Alok and the rest of the tiering team to confirm I am not misunderstanding your question. 

Assuming we are level-set on what Alok refers to, you could add text to the effect: "When attach tier starts, an internal process called fix-layout commences internally to prepare the hot tier for use. This preparation may take some time. Once it completes normal tiering activities shall begin. This administrator should be aware that this may delay the start of tiering activities."

Comment 18 Nag Pavan Chilakam 2016-01-27 07:34:06 UTC

Divya,
Do you want to add the above comment in doc?

Comment 19 Divya 2016-01-27 09:40:56 UTC

(In reply to nchilaka from comment #12)
> Following are my comments for 11.7. Detaching a Tier from a Volume section:
> 
> Comment#1:
> 
> Step#1:gluster volume detach-tier VOLNAME start
> 
> Comment:I thought we would be using new cli ie, gluster volume detach
> VOLNAME tier start "eg:gluster v tier countme detach  start"

I have updated the command.
> 
> 
> Comment#2:
> Step#2 
> comment: same as above, we should be using "gluster v tier countme detach
> status"

I have updated the command.

> 
> Comment#3:
> ==> Same to be implemented for detach tier in Geo-rep session too"

I have updated the command.
> 
> 
> 
> Comment#4:
> 10.3.1.1. Expanding a Cold Tier Volume--> I feel it is incomplete. We are
> telling admin to detach tier, add new bricks.
> We should also tell him to reattach the tier, using normal tier attach
> command
> 
Added a step to reattach the tier.
> 
> 
> Comment#5:
> We must be putting a note or somekind of highlighting done , for both
> expanding and shrinking tier volumes, saying " it is a disruptive opeation"

Added "When you reattach a tier, an internal process called fix-layout commences internally to prepare the hot tier for use. This process takes time and there will a delay in starting the tiering activities. " in the respective sections as a Important note.

Link to the Expanding Volumes section: http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20(html-single)/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Red_Hat_Storage_Volumes-Expanding_Tiered_Volume

Link to the Shrinking Volumes section: http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20(html-single)/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Red_Hat_Storage_Volumes-Shrinking_Tiered_Volume

Link to the Detach Tier section: http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20%28html-single%29/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Data_Tieirng-Detach_Tier

Comment 20 Dan Lambright 2016-01-27 12:41:06 UTC

10.3.1.1 

First paragraph, change "increases the" to "increase the"

Otherwise looks great.

Comment 21 Nag Pavan Chilakam 2016-01-28 05:44:18 UTC

based on comment 20, moving it to verified

Comment 22 Divya 2016-01-28 06:50:20 UTC

(In reply to Dan Lambright from comment #20)
> 10.3.1.1 
> 
> First paragraph, change "increases the" to "increase the"
> 
> Otherwise looks great.

Fixed the typo. Link to the latest doc: http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20%28html-single%29/lastStableBuild/artifact/tmp/en-US/html-single/index.html#chap-Managing_Red_Hat_Storage_Volumes-Expanding_Tiered_Volume