Description of problem: After Performing volume expansion on an fully utilized volume getting a no space left on device error. Version-Release number of selected component (if applicable): heketi-client-5.0.0-6.el7rhgs.x86_64 cns-deploy-5.0.0-12.el7rhgs.x86_64 How reproducible: Always Steps to Reproduce: 1. Fill a Volume with its full storage capacity 2.Perform a Volume expand through heketi on the same volume 3.Try to Write a new file /dir inside the volume Actual results: Cannot create File / Dir with the following error No space left on device Expected results: Should be able to write on newly expanded volume Additional info: sh-4.3# df -kh Filesystem Size Used Avail Use% Mounted on /dev/mapper/docker-253:3-100718180-fa8613dfd8bbcbe28eff7f3b2e5784419821d4128c5139fda12cff00d16e03ad 10G 643M 9.4G 7% / tmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59G 511M 59G 1% /etc/hosts shm 64M 0 64M 0% /dev/shm 10.70.46.163:vol_126a86dda5cc5d24b021ea6e77e85ba8 5.0G 4.0G 984M 81% /var/www/html tmpfs 3.9G 16K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount sh-4.3# pwd /var/www/html sh-4.3# mkdir test1 mkdir: cannot create directory 'test1': No space left on device [root@dhcp46-169 ~]# heketi-cli volume info 126a86dda5cc5d24b021ea6e77e85ba8 Name: vol_126a86dda5cc5d24b021ea6e77e85ba8 Size: 5 Volume Id: 126a86dda5cc5d24b021ea6e77e85ba8 Cluster Id: 63fb952cf06f0e1cb7a49e24c780bdd4 Mount: 10.70.46.163:vol_126a86dda5cc5d24b021ea6e77e85ba8 Mount Options: backup-volfile-servers=10.70.47.188,10.70.47.44 Block: false Free Size: 0 Block Volumes: [] Durability Type: replicate Distributed+Replica: 3 [root@dhcp46-169 ~]#
As per Humble's suggestion i ran a manual re-balance of the volume from the gluster pod and after that i am able to utilize the expanded storage. But the volume re-balance should be happening automatically as a part of volume expansion operation which is not the case as of now
With new builds i am able to perform a volume expansion and write data onto new bricks without doing a manual re-balance Performed following scenarios 1. Performed Volume expansion, created a new file and directory and perform I/O inside existing directories 2. Created new directories under the mount point and performed I/O. 3. Performed a volume expansion when the volume is full and performed I/O 4. Performed a volume expansion before volume is full and performed I/O Following builds were used for the verification heketi-client-5.0.0-7.el7rhgs.x86_64 cns-deploy-5.0.0-14.el7rhgs.x86_64 Gluster - rhgs-server-rhel7:3.3.0-11 Heketi - rhgs-volmanager-rhel7:3.3.0-9
Thanks a lot Tejus for the update!!
doc text looks good to me
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:2879