Bug 1477431

Summary: After Volume Expansion is completed successfully getting write error inside the volume
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Tejas Chaphekar <tchaphek>
Component: heketiAssignee: Mohamed Ashiq <mliyazud>
Status: CLOSED ERRATA QA Contact: Tejas Chaphekar <tchaphek>
Severity: high Docs Contact:
Priority: unspecified    
Version: cns-3.6CC: asriram, hchiramm, madam, mliyazud, pprakash, rhs-bugs, rtalur, srmukher, storage-qa-internal, vinug
Target Milestone: ---   
Target Release: CNS 3.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: heketi-5.0.0-7 rhgs-volmanager-docker-5.0.0-9 Doc Type: Bug Fix
Doc Text:
Prior to this update, after expansion of a volume using heketi-cli, rebalance of volume was not performed which led to write errors on files that resided on old bricks. With this fix, a rebalance operation is initiated on the volume after every expansion.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-11 07:09:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1440900, 1445448    

Description Tejas Chaphekar 2017-08-02 04:55:06 UTC
Description of problem:

After Performing volume expansion on an fully utilized volume getting a no space left on device error.

Version-Release number of selected component (if applicable):
heketi-client-5.0.0-6.el7rhgs.x86_64
cns-deploy-5.0.0-12.el7rhgs.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Fill a Volume with its full storage capacity 
2.Perform a Volume expand through heketi on the same volume
3.Try to Write a new file /dir inside the volume

Actual results:

Cannot create File / Dir with the following error
No space left on device


Expected results:

Should be able to write on newly expanded volume

Additional info:


sh-4.3# df -kh
Filesystem                                                                                           Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:3-100718180-fa8613dfd8bbcbe28eff7f3b2e5784419821d4128c5139fda12cff00d16e03ad   10G  643M  9.4G   7% /
tmpfs                                                                                                3.9G     0  3.9G   0% /dev
tmpfs                                                                                                3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var                                                         59G  511M   59G   1% /etc/hosts
shm                                                                                                   64M     0   64M   0% /dev/shm
10.70.46.163:vol_126a86dda5cc5d24b021ea6e77e85ba8                                                    5.0G  4.0G  984M  81% /var/www/html
tmpfs                                                                                                3.9G   16K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
sh-4.3# pwd
/var/www/html
sh-4.3# mkdir test1
mkdir: cannot create directory 'test1': No space left on device


[root@dhcp46-169 ~]# heketi-cli volume info 126a86dda5cc5d24b021ea6e77e85ba8
Name: vol_126a86dda5cc5d24b021ea6e77e85ba8
Size: 5
Volume Id: 126a86dda5cc5d24b021ea6e77e85ba8
Cluster Id: 63fb952cf06f0e1cb7a49e24c780bdd4
Mount: 10.70.46.163:vol_126a86dda5cc5d24b021ea6e77e85ba8
Mount Options: backup-volfile-servers=10.70.47.188,10.70.47.44
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3
[root@dhcp46-169 ~]#

Comment 2 Tejas Chaphekar 2017-08-03 06:54:23 UTC
As per Humble's suggestion i ran a manual re-balance of the volume from the gluster pod and after that i am able to utilize the expanded storage.               

But the volume re-balance should be happening automatically as a part of volume expansion operation which is not the case as of now

Comment 7 Tejas Chaphekar 2017-08-10 09:39:18 UTC
With new builds i am able to perform a volume expansion and write data onto new bricks without doing a manual re-balance

Performed following scenarios

1. Performed Volume expansion, created a new file and directory and perform I/O inside existing directories

2. Created new directories under the mount point and performed I/O.

3. Performed a volume expansion when the volume is full and performed I/O

4. Performed a volume expansion before volume is full and performed I/O

Following builds were used for the verification

heketi-client-5.0.0-7.el7rhgs.x86_64
cns-deploy-5.0.0-14.el7rhgs.x86_64
Gluster - rhgs-server-rhel7:3.3.0-11
Heketi -  rhgs-volmanager-rhel7:3.3.0-9

Comment 8 Tejas Chaphekar 2017-08-10 09:39:58 UTC
With new builds i am able to perform a volume expansion and write data onto new bricks without doing a manual re-balance

Performed following scenarios

1. Performed Volume expansion, created a new file and directory and perform I/O inside existing directories

2. Created new directories under the mount point and performed I/O.

3. Performed a volume expansion when the volume is full and performed I/O

4. Performed a volume expansion before volume is full and performed I/O

Following builds were used for the verification

heketi-client-5.0.0-7.el7rhgs.x86_64
cns-deploy-5.0.0-14.el7rhgs.x86_64
Gluster - rhgs-server-rhel7:3.3.0-11
Heketi -  rhgs-volmanager-rhel7:3.3.0-9

Comment 9 Humble Chirammal 2017-08-10 09:58:14 UTC
Thanks a lot Tejus for the update!!

Comment 11 Raghavendra Talur 2017-10-04 15:39:20 UTC
doc text looks good to me

Comment 12 errata-xmlrpc 2017-10-11 07:09:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2879