Description of problem: When a volume is expanded via heketi using volume expand option, the additional capacity is not usable although the size is available i.e., Rebalance doesn't seem to be happening. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. created a 5 gb volume heketi-cli volume info 4b26cd0b5ad13a5e606734bc7a8b8d2d Name: vol_4b26cd0b5ad13a5e606734bc7a8b8d2d Size: 10 Volume Id: 4b26cd0b5ad13a5e606734bc7a8b8d2d Cluster Id: e32cd6139d9acb83d110f2c53b2a7359 Mount: 10.70.47.98:vol_4b26cd0b5ad13a5e606734bc7a8b8d2d Mount Options: backup-volfile-servers=10.70.47.1,10.70.46.217,10.70.46.67,10.70.46.171,10.70.47.182 Block: false Free Size: 0 Block Volumes: [] Durability Type: replicate Distributed+Replica: 3 2. Mounted the volume from one of the node and wrote 3 gb of data 3. expanded the volume with another 5 gb (now the total size of volume is 10gb) 4. tried writing another 5gb of data from the mountpoint, however file creation failed after reaching 5gb. Actual results: expanded capacity is not usable Expected results: expanded capacity should be consumable Additional info: Deployment was done via ansible heketi logs shall be attached
Created attachment 1420888 [details] heketi_logs
Madhu and I debugged it. It appears that the heketi.json template that is in cns-deploy[1] has the parameter to set rebalance_on_expansion to true whereas the ansible installer[2] does not. This is also the reason why we could hit the bug in CNS 3.9.0 testing. That particular setup was a cns-deploy created one. Fix is to make the default in heketi binary as true and only disable rebalance on expansion if parameter is set to false. [1] https://github.com/gluster/gluster-kubernetes/blob/master/deploy/heketi.json.template [2] https://github.com/openshift/openshift-ansible/blob/e6a94e625ec68300b571a09318fdf90e61c33380/roles/openshift_storage_glusterfs/templates/heketi.json.j2
With rhgs-volmanager-container-3.3.1-8.4, expanded capacity is consumable. Repeated the test mentioedn in steps to reproduce and volume expansion works as expected. Moving the bug to verified.
The line "With this fix, the option has been set to true by default in the template and rebalance operation is performed on every volume expansion operation." needs to be changed a little the change in default is in heketi binary, hence we should either omit specifying where the default has changed or tell that the change in heketi binary.
(In reply to Raghavendra Talur from comment #7) > The line > "With this fix, the option has been set to true by default in the template > and rebalance operation is performed on every volume expansion operation." > > needs to be changed a little > > the change in default is in heketi binary, hence we should either omit > specifying where the default has changed or tell that the change in heketi > binary. Proposed change: With this fix, the option is set to true by default in the heketi binary template and rebalance operation is performed on every volume expansion operation successfully. Please review and ack.
Proposed change: With this fix, the option is set to true by default in the heketi binary and rebalance operation is performed on every volume expansion operation successfully. I just removed the word "template" from the line.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1178