Bug 1566590

Summary: expanded capacity not usable after expanding a volume using heketi
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: krishnaram Karthick <kramdoss>
Component: heketiAssignee: Michael Adam <madam>
Status: CLOSED ERRATA QA Contact: krishnaram Karthick <kramdoss>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: cns-3.9CC: asriram, hchiramm, pprakash, rcyriac, rhs-bugs, rtalur, storage-qa-internal, vinug
Target Milestone: ---Keywords: ZStream
Target Release: CNS 3.9 Async   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhgs-volmanager-container-3.3.1-8.4 Doc Type: Bug Fix
Doc Text:
Previously, in CNS installations deployed using ansible installer, rebalance operation was not executed after a volume expansion operation. This was due to the environment variable missing from the template which sets the option to true. With this fix, the option is set to true by default in the heketi binary and rebalance operation is performed on every volume expansion operation.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-19 03:34:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
heketi_logs none

Description krishnaram Karthick 2018-04-12 14:59:55 UTC 
Description of problem:
When a volume is expanded via heketi using volume expand option, the additional capacity is not usable although the size is available i.e., Rebalance doesn't seem to be happening.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. created a 5 gb volume 

heketi-cli volume info 4b26cd0b5ad13a5e606734bc7a8b8d2d
Name: vol_4b26cd0b5ad13a5e606734bc7a8b8d2d
Size: 10
Volume Id: 4b26cd0b5ad13a5e606734bc7a8b8d2d
Cluster Id: e32cd6139d9acb83d110f2c53b2a7359
Mount: 10.70.47.98:vol_4b26cd0b5ad13a5e606734bc7a8b8d2d
Mount Options: backup-volfile-servers=10.70.47.1,10.70.46.217,10.70.46.67,10.70.46.171,10.70.47.182
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

2. Mounted the volume from one of the node and wrote 3 gb of data
3. expanded the volume with another 5 gb (now the total size of volume is 10gb)
4. tried writing another 5gb of data from the mountpoint, however file creation failed after reaching 5gb.

Actual results:
expanded capacity is not usable

Expected results:
expanded capacity should be consumable

Additional info:
Deployment was done via ansible

heketi logs shall be attached
Comment 2 krishnaram Karthick 2018-04-12 15:03:00 UTC 
Created attachment 1420888 [details]
heketi_logs
Comment 3 Raghavendra Talur 2018-04-12 15:06:13 UTC 
Madhu and I debugged it.

It appears that the heketi.json template that is in cns-deploy[1] has the parameter to set rebalance_on_expansion to true whereas the ansible installer[2] does not. 

This is also the reason why we could hit the bug in CNS 3.9.0 testing. That particular setup was a cns-deploy created one.

Fix is to make the default in heketi binary as true and only disable rebalance on expansion if parameter is set to false.

[1] https://github.com/gluster/gluster-kubernetes/blob/master/deploy/heketi.json.template
[2] https://github.com/openshift/openshift-ansible/blob/e6a94e625ec68300b571a09318fdf90e61c33380/roles/openshift_storage_glusterfs/templates/heketi.json.j2
Comment 5 krishnaram Karthick 2018-04-16 10:57:33 UTC 
With rhgs-volmanager-container-3.3.1-8.4, expanded capacity is consumable. 

Repeated the test mentioedn in steps to reproduce and volume expansion works as expected.

Moving the bug to verified.
Comment 7 Raghavendra Talur 2018-04-17 06:07:33 UTC 
The line 
"With this fix, the option has been set to true by default in the template and rebalance operation is performed on every volume expansion operation."

needs to be changed a little

the change in default is in heketi binary, hence we should either omit specifying where the default has changed or tell that the change in heketi binary.
Comment 8 Anjana Suparna Sriram 2018-04-18 08:17:56 UTC 
(In reply to Raghavendra Talur from comment #7)
> The line 
> "With this fix, the option has been set to true by default in the template
> and rebalance operation is performed on every volume expansion operation."
> 
> needs to be changed a little
> 
> the change in default is in heketi binary, hence we should either omit
> specifying where the default has changed or tell that the change in heketi
> binary.

Proposed change: With this fix, the option is set to true by default in the heketi binary template and rebalance operation is performed on every volume expansion operation successfully.

Please review and ack.
Comment 9 Raghavendra Talur 2018-04-18 13:08:28 UTC 
Proposed change: With this fix, the option is set to true by default in the heketi binary and rebalance operation is performed on every volume expansion operation successfully.


I just removed the word "template" from the line.
Comment 12 errata-xmlrpc 2018-04-19 03:34:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1178