Red Hat Bugzilla – Bug 1477431
After Volume Expansion is completed successfully getting write error inside the volume
Last modified: 2017-10-11 03:09:46 EDT
Description of problem:
After Performing volume expansion on an fully utilized volume getting a no space left on device error.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Fill a Volume with its full storage capacity
2.Perform a Volume expand through heketi on the same volume
3.Try to Write a new file /dir inside the volume
Cannot create File / Dir with the following error
No space left on device
Should be able to write on newly expanded volume
sh-4.3# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-253:3-100718180-fa8613dfd8bbcbe28eff7f3b2e5784419821d4128c5139fda12cff00d16e03ad 10G 643M 9.4G 7% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59G 511M 59G 1% /etc/hosts
shm 64M 0 64M 0% /dev/shm
10.70.46.163:vol_126a86dda5cc5d24b021ea6e77e85ba8 5.0G 4.0G 984M 81% /var/www/html
tmpfs 3.9G 16K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount
sh-4.3# mkdir test1
mkdir: cannot create directory 'test1': No space left on device
[root@dhcp46-169 ~]# heketi-cli volume info 126a86dda5cc5d24b021ea6e77e85ba8
Volume Id: 126a86dda5cc5d24b021ea6e77e85ba8
Cluster Id: 63fb952cf06f0e1cb7a49e24c780bdd4
Mount Options: backup-volfile-servers=10.70.47.188,10.70.47.44
Free Size: 0
Block Volumes: 
Durability Type: replicate
As per Humble's suggestion i ran a manual re-balance of the volume from the gluster pod and after that i am able to utilize the expanded storage.
But the volume re-balance should be happening automatically as a part of volume expansion operation which is not the case as of now
With new builds i am able to perform a volume expansion and write data onto new bricks without doing a manual re-balance
Performed following scenarios
1. Performed Volume expansion, created a new file and directory and perform I/O inside existing directories
2. Created new directories under the mount point and performed I/O.
3. Performed a volume expansion when the volume is full and performed I/O
4. Performed a volume expansion before volume is full and performed I/O
Following builds were used for the verification
Gluster - rhgs-server-rhel7:3.3.0-11
Heketi - rhgs-volmanager-rhel7:3.3.0-9
Thanks a lot Tejus for the update!!
doc text looks good to me
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.