Bug 1535332 - The used space in the volume increases when the volume is expanded
Summary: The used space in the volume increases when the volume is expanded
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhgs-server-container
Version: cns-3.6
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Raghavendra Talur
QA Contact: Rachael
URL:
Whiteboard:
Depends On: 1540961
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-17 07:11 UTC by Rachael
Modified: 2019-12-09 02:21 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1540961 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Rachael 2018-01-17 07:11:11 UTC
Description of problem:

A volume of size 40GB was created and a file of 39GB was created on it. The volume was expanded by 30GB and the total size of the volume became 70GB. But the used space on the volume increased from 39GB to 61.5GB

Version-Release number of selected component (if applicable):

cns-deploy-5.0.0-57.el7rhgs.x86_64
heketi-client-5.0.0-19.el7rhgs.x86_64

How reproducible:


Steps to Reproduce:
1.[root@dhcp46-2 ~]# oc rsh busybox
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687
                         10.0G     34.1M      9.9G   0% /
tmpfs                    11.7G         0     11.7G   0% /dev
tmpfs                    11.7G         0     11.7G   0% /sys/fs/cgroup
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /dev/termination-log
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /run/secrets
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /etc/resolv.conf
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /etc/hostname
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad
                         40.0G     33.1M     39.9G   0% /usr/share/busybox
tmpfs                    11.7G     16.0K     11.7G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    11.7G         0     11.7G   0% /proc/kcore
tmpfs                    11.7G         0     11.7G   0% /proc/timer_list
tmpfs                    11.7G         0     11.7G   0% /proc/timer_stats
tmpfs                    11.7G         0     11.7G   0% /proc/sched_debug

2. / # cd /usr/share/busybox
/usr/share/busybox # fallocate -l 39G 39gfile

3./usr/share/busybox # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687
                         10.0G     34.1M      9.9G   0% /
tmpfs                    11.7G         0     11.7G   0% /dev
tmpfs                    11.7G         0     11.7G   0% /sys/fs/cgroup
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /dev/termination-log
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /run/secrets
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /etc/resolv.conf
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /etc/hostname
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.4M     58.3G   1% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad
                         40.0G     39.0G    966.9M  98% /usr/share/busybox
tmpfs                    11.7G     16.0K     11.7G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    11.7G         0     11.7G   0% /proc/kcore
tmpfs                    11.7G         0     11.7G   0% /proc/timer_list
tmpfs                    11.7G         0     11.7G   0% /proc/timer_stats
tmpfs                    11.7G         0     11.7G   0% /proc/sched_debug

4. [root@dhcp46-2 ~]# heketi-cli volume expand --expand-size=30 --volume=14a05e3c1f728afdebc484c15f7c6cad
Name: vol_14a05e3c1f728afdebc484c15f7c6cad
Size: 70
Volume Id: 14a05e3c1f728afdebc484c15f7c6cad
Cluster Id: 096f8c70caa54788a00e4458c052656e
Mount: 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad
Mount Options: backup-volfile-servers=10.70.46.21,10.70.47.68
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

5. / # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687
                         10.0G     34.1M      9.9G   0% /
tmpfs                    11.7G         0     11.7G   0% /dev
tmpfs                    11.7G         0     11.7G   0% /sys/fs/cgroup
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.7M     58.3G   1% /dev/termination-log
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.7M     58.3G   1% /run/secrets
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.7M     58.3G   1% /etc/resolv.conf
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.7M     58.3G   1% /etc/hostname
/dev/mapper/vg_rhel_dhcp47----104--var-lv_var
                         59.0G    642.7M     58.3G   1% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad
                         70.0G     61.5G      8.4G  88% /usr/share/busybox
tmpfs                    11.7G     16.0K     11.7G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    11.7G         0     11.7G   0% /proc/kcore
tmpfs                    11.7G         0     11.7G   0% /proc/timer_list
tmpfs                    11.7G         0     11.7G   0% /proc/timer_stats
tmpfs                    11.7G         0     11.7G   0% /proc/sched_debug


Actual results:
The used space increases to 61.5GB

Expected results:
The used space should stay the same i.e., 39GB

Additional info:

Comment 2 Raghavendra Talur 2018-01-18 07:43:01 UTC
** gluster volume info after expansion:
sh-4.2# gluster vol info vol_14a05e3c1f728afdebc484c15f7c6cad                  
                                       
Volume Name: vol_14a05e3c1f728afdebc484c15f7c6cad                              
Type: Distributed-Replicate            
Volume ID: 15550f1f-743b-494d-b761-81d03c8bb24f                                
Status: Started                        
Snapshot Count: 0                      
Number of Bricks: 2 x 3 = 6            
Transport-type: tcp                    
Bricks:                                
Brick1: 10.70.46.21:/var/lib/heketi/mounts/vg_02b16cdadce2aaf1356c37599dadf3f4/brick_a99f4919f14c0ab20b667a0269adeff5/brick                                   
Brick2: 10.70.47.68:/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_1b0082ad01b71b87e16686c5667cab6e/brick                                   
Brick3: 10.70.46.153:/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_35a93294d10a783faf85aca0d8ced048/brick                                  
Brick4: 10.70.46.153:/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_7542d4af880f8f2b501180e0a2e1f117/brick                                  
Brick5: 10.70.46.21:/var/lib/heketi/mounts/vg_bf6b608031932669bb05cc8430948619/brick_490a6de2e0e5b67e7c9495b3d57bb9e0/brick                                   
Brick6: 10.70.47.68:/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_7c97d7dc6c2f0a3d77bffd3e2c95bd7e/brick                                   
Options Reconfigured:                  
nfs.disable: on                        
transport.address-family: inet         
cluster.brick-multiplex: on      



** ls on the client after expansion:
# ls -lash /usr/share/busybox/
total 40894472                          
     4 drwxrwsr-x    4 root     2000        4.0K Jan 16 14:25 .                 
     0 drwxr-xr-x    3 root     root          21 Jan 18 07:12 ..                
     4 drwxr-sr-x    3 root     2000        4.0K Jan 16 14:25 .trashcan         
40894464 -rw-r--r--    1 root     2000       39.0G Jan 16 14:23 39gfile         

** ls on the bricks after expansion
* node 1
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_7c97d7dc6c2f0a3d77bffd3e2c95bd7e/brick/                                     
total 23G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:25 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:25 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
23G ---------T. 2 root 2000   0 Jan 16 14:25 39gfile                           
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_1b0082ad01b71b87e16686c5667cab6e/brick/                                     
total 40G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:23 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:14 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile    

* node 2
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_02b16cdadce2aaf1356c37599dadf3f4/brick_a99f4919f14c0ab20b667a0269adeff5/brick/                                     
total 40G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:23 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:14 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile                           
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_bf6b608031932669bb05cc8430948619/brick_490a6de2e0e5b67e7c9495b3d57bb9e0/brick/                                     
total 23G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:25 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:25 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
23G ---------T. 2 root 2000   0 Jan 16 14:25 39gfile         

* node 3
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_35a93294d10a783faf85aca0d8ced048/brick/                                     
total 40G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:23 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:14 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile                           
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_7542d4af880f8f2b501180e0a2e1f117/brick/                                     
total 23G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:25 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:25 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
23G ---------T. 2 root 2000   0 Jan 16 14:25 39gfile

Comment 3 Raghavendra Talur 2018-01-18 07:51:19 UTC
df output seems to showing sum of space occupied by real file and link to file 

10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad
                         70.0G     61.5G      8.4G  88% /usr/share/busybox

Comment 10 Rachael 2018-01-31 06:31:17 UTC
The system details are : 
OCP master 10.70.46.2 root/aplo

Comment 11 Nithya Balachandran 2018-02-01 09:23:03 UTC
This is probably the root cause (found by vbellur): 

https://stackoverflow.com/questions/32799252/why-fallocate-on-linux-creates-a-non-empty-file-when-it-has-not-enough-space

Comment 12 Nithya Balachandran 2018-02-01 09:39:28 UTC
https://code.engineering.redhat.com/gerrit/#/c/129236/1 tests the hypothesis. If this works, I will redo this cleanly.

Comment 13 Raghavendra Talur 2018-02-01 12:20:20 UTC
This bug is now a tracker for RHGS bug 1540961


Note You need to log in before you can comment on or make changes to this bug.