Bug 1566590 - expanded capacity not usable after expanding a volume using heketi
Summary: expanded capacity not usable after expanding a volume using heketi
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: cns-3.9
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: CNS 3.9 Async
Assignee: Michael Adam
QA Contact: krishnaram Karthick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-12 14:59 UTC by krishnaram Karthick
Modified: 2018-11-17 08:34 UTC (History)
8 users (show)

Fixed In Version: rhgs-volmanager-container-3.3.1-8.4
Doc Type: Bug Fix
Doc Text:
Previously, in CNS installations deployed using ansible installer, rebalance operation was not executed after a volume expansion operation. This was due to the environment variable missing from the template which sets the option to true. With this fix, the option is set to true by default in the heketi binary and rebalance operation is performed on every volume expansion operation.
Clone Of:
Environment:
Last Closed: 2018-04-19 03:34:39 UTC
Target Upstream Version:


Attachments (Terms of Use)
heketi_logs (1.81 MB, text/plain)
2018-04-12 15:03 UTC, krishnaram Karthick
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1178 0 None None None 2018-04-19 03:34:55 UTC

Description krishnaram Karthick 2018-04-12 14:59:55 UTC
Description of problem:
When a volume is expanded via heketi using volume expand option, the additional capacity is not usable although the size is available i.e., Rebalance doesn't seem to be happening.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. created a 5 gb volume 

heketi-cli volume info 4b26cd0b5ad13a5e606734bc7a8b8d2d
Name: vol_4b26cd0b5ad13a5e606734bc7a8b8d2d
Size: 10
Volume Id: 4b26cd0b5ad13a5e606734bc7a8b8d2d
Cluster Id: e32cd6139d9acb83d110f2c53b2a7359
Mount: 10.70.47.98:vol_4b26cd0b5ad13a5e606734bc7a8b8d2d
Mount Options: backup-volfile-servers=10.70.47.1,10.70.46.217,10.70.46.67,10.70.46.171,10.70.47.182
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

2. Mounted the volume from one of the node and wrote 3 gb of data
3. expanded the volume with another 5 gb (now the total size of volume is 10gb)
4. tried writing another 5gb of data from the mountpoint, however file creation failed after reaching 5gb.

Actual results:
expanded capacity is not usable

Expected results:
expanded capacity should be consumable

Additional info:
Deployment was done via ansible

heketi logs shall be attached

Comment 2 krishnaram Karthick 2018-04-12 15:03:00 UTC
Created attachment 1420888 [details]
heketi_logs

Comment 3 Raghavendra Talur 2018-04-12 15:06:13 UTC
Madhu and I debugged it.

It appears that the heketi.json template that is in cns-deploy[1] has the parameter to set rebalance_on_expansion to true whereas the ansible installer[2] does not. 

This is also the reason why we could hit the bug in CNS 3.9.0 testing. That particular setup was a cns-deploy created one.

Fix is to make the default in heketi binary as true and only disable rebalance on expansion if parameter is set to false.

[1] https://github.com/gluster/gluster-kubernetes/blob/master/deploy/heketi.json.template
[2] https://github.com/openshift/openshift-ansible/blob/e6a94e625ec68300b571a09318fdf90e61c33380/roles/openshift_storage_glusterfs/templates/heketi.json.j2

Comment 5 krishnaram Karthick 2018-04-16 10:57:33 UTC
With rhgs-volmanager-container-3.3.1-8.4, expanded capacity is consumable. 

Repeated the test mentioedn in steps to reproduce and volume expansion works as expected.

Moving the bug to verified.

Comment 7 Raghavendra Talur 2018-04-17 06:07:33 UTC
The line 
"With this fix, the option has been set to true by default in the template and rebalance operation is performed on every volume expansion operation."

needs to be changed a little

the change in default is in heketi binary, hence we should either omit specifying where the default has changed or tell that the change in heketi binary.

Comment 8 Anjana Suparna Sriram 2018-04-18 08:17:56 UTC
(In reply to Raghavendra Talur from comment #7)
> The line 
> "With this fix, the option has been set to true by default in the template
> and rebalance operation is performed on every volume expansion operation."
> 
> needs to be changed a little
> 
> the change in default is in heketi binary, hence we should either omit
> specifying where the default has changed or tell that the change in heketi
> binary.

Proposed change: With this fix, the option is set to true by default in the heketi binary template and rebalance operation is performed on every volume expansion operation successfully.

Please review and ack.

Comment 9 Raghavendra Talur 2018-04-18 13:08:28 UTC
Proposed change: With this fix, the option is set to true by default in the heketi binary and rebalance operation is performed on every volume expansion operation successfully.


I just removed the word "template" from the line.

Comment 12 errata-xmlrpc 2018-04-19 03:34:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1178


Note You need to log in before you can comment on or make changes to this bug.