Bug 1477431 - After Volume Expansion is completed successfully getting write error inside the volume
After Volume Expansion is completed successfully getting write error inside t...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi (Show other bugs)
cns-3.6
Unspecified Unspecified
unspecified Severity high
: ---
: CNS 3.6
Assigned To: Mohamed Ashiq
Tejas Chaphekar
:
Depends On:
Blocks: 1440900 1445448
  Show dependency treegraph
 
Reported: 2017-08-02 00:55 EDT by Tejas Chaphekar
Modified: 2017-10-11 03:09 EDT (History)
9 users (show)

See Also:
Fixed In Version: heketi-5.0.0-7 rhgs-volmanager-docker-5.0.0-9
Doc Type: Bug Fix
Doc Text:
Prior to this update, after expansion of a volume using heketi-cli, rebalance of volume was not performed which led to write errors on files that resided on old bricks. With this fix, a rebalance operation is initiated on the volume after every expansion.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-11 03:09:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Tejas Chaphekar 2017-08-02 00:55:06 EDT
Description of problem:

After Performing volume expansion on an fully utilized volume getting a no space left on device error.

Version-Release number of selected component (if applicable):
heketi-client-5.0.0-6.el7rhgs.x86_64
cns-deploy-5.0.0-12.el7rhgs.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Fill a Volume with its full storage capacity 
2.Perform a Volume expand through heketi on the same volume
3.Try to Write a new file /dir inside the volume

Actual results:

Cannot create File / Dir with the following error
No space left on device


Expected results:

Should be able to write on newly expanded volume

Additional info:


sh-4.3# df -kh
Filesystem                                                                                           Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:3-100718180-fa8613dfd8bbcbe28eff7f3b2e5784419821d4128c5139fda12cff00d16e03ad   10G  643M  9.4G   7% /
tmpfs                                                                                                3.9G     0  3.9G   0% /dev
tmpfs                                                                                                3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var                                                         59G  511M   59G   1% /etc/hosts
shm                                                                                                   64M     0   64M   0% /dev/shm
10.70.46.163:vol_126a86dda5cc5d24b021ea6e77e85ba8                                                    5.0G  4.0G  984M  81% /var/www/html
tmpfs                                                                                                3.9G   16K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
sh-4.3# pwd
/var/www/html
sh-4.3# mkdir test1
mkdir: cannot create directory 'test1': No space left on device


[root@dhcp46-169 ~]# heketi-cli volume info 126a86dda5cc5d24b021ea6e77e85ba8
Name: vol_126a86dda5cc5d24b021ea6e77e85ba8
Size: 5
Volume Id: 126a86dda5cc5d24b021ea6e77e85ba8
Cluster Id: 63fb952cf06f0e1cb7a49e24c780bdd4
Mount: 10.70.46.163:vol_126a86dda5cc5d24b021ea6e77e85ba8
Mount Options: backup-volfile-servers=10.70.47.188,10.70.47.44
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3
[root@dhcp46-169 ~]#
Comment 2 Tejas Chaphekar 2017-08-03 02:54:23 EDT
As per Humble's suggestion i ran a manual re-balance of the volume from the gluster pod and after that i am able to utilize the expanded storage.               

But the volume re-balance should be happening automatically as a part of volume expansion operation which is not the case as of now
Comment 7 Tejas Chaphekar 2017-08-10 05:39:18 EDT
With new builds i am able to perform a volume expansion and write data onto new bricks without doing a manual re-balance

Performed following scenarios

1. Performed Volume expansion, created a new file and directory and perform I/O inside existing directories

2. Created new directories under the mount point and performed I/O.

3. Performed a volume expansion when the volume is full and performed I/O

4. Performed a volume expansion before volume is full and performed I/O

Following builds were used for the verification

heketi-client-5.0.0-7.el7rhgs.x86_64
cns-deploy-5.0.0-14.el7rhgs.x86_64
Gluster - rhgs-server-rhel7:3.3.0-11
Heketi -  rhgs-volmanager-rhel7:3.3.0-9
Comment 8 Tejas Chaphekar 2017-08-10 05:39:58 EDT
With new builds i am able to perform a volume expansion and write data onto new bricks without doing a manual re-balance

Performed following scenarios

1. Performed Volume expansion, created a new file and directory and perform I/O inside existing directories

2. Created new directories under the mount point and performed I/O.

3. Performed a volume expansion when the volume is full and performed I/O

4. Performed a volume expansion before volume is full and performed I/O

Following builds were used for the verification

heketi-client-5.0.0-7.el7rhgs.x86_64
cns-deploy-5.0.0-14.el7rhgs.x86_64
Gluster - rhgs-server-rhel7:3.3.0-11
Heketi -  rhgs-volmanager-rhel7:3.3.0-9
Comment 9 Humble Chirammal 2017-08-10 05:58:14 EDT
Thanks a lot Tejus for the update!!
Comment 11 Raghavendra Talur 2017-10-04 11:39:20 EDT
doc text looks good to me
Comment 12 errata-xmlrpc 2017-10-11 03:09:46 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2879

Note You need to log in before you can comment on or make changes to this bug.