Bug 764381 (GLUSTER-2649)

Summary: Quota: size field is not getting updated properly
Product: [Community] GlusterFS Reporter: Saurabh <saurabh>
Component: quotaAssignee: Raghavendra G <raghavendra>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: mainlineCC: gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Saurabh 2011-04-01 11:13:40 UTC 
Hello,


  I had volume with 2MB limit set and a directory with in it having 5MB limit set.

Now , over fuse mount when some files are created the size shown is this,


gluster> volume quota dist2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152              1546240
/d1                     5242880              1546240
/d2                     5242880                    0

Again, after removing all the files and making new files provides me size as,
gluster> volume quota dist2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152              1515520
/d1                     5242880              1515520
/d2                     5242880                    0

but again for validation all files were removed,
and new files were created,

gluster> volume quota dist2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152                    0
/d1                     5242880                    0
/d2                     5242880                    0
gluster> 
gluster> 
gluster> volume quota dist2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152              1612800
/d1                     5242880              1612800
/d2                     5242880                    0


Point here is that there is a mismatch between the amount of data collected from client side and server side, after all this exercise.


after the last trial the client(over fuse mount) says the total size as 

[root@centos-qa-client-1 d1]# bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
5120*205
1049600
./1024
1025
./1024
1


and the extended attributes cumulatively show the same on servers.

but the size displayed by the "list" command is wrong.



Also, as per my observation for the last trial, when all files were removed and newly created just occupied 1MB(approx), the "Disk quota exceeded" message starts appearing, whereas the volume space limit is 2MB and it is just double of the size allowed to be occupied by new files. 

So, this error brings a confusion about how much before the space-quota getting filled the Warning for "Disk Quota exceeded" kind of message will start appearing?
Comment 1 Junaid 2011-04-05 08:00:53 UTC 
*** Bug 2671 has been marked as a duplicate of this bug. ***
Comment 2 Saurabh 2011-04-07 07:20:39 UTC 
[root@centos-qa-client-1 d1]# bc 
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
1023*1024
1047552



is the total size of the 1023 files created under a dir over nfs mount.

The same size is updated using the xattr on the backend.



But the size field is again not updated properly.
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dist2
getfattr: Removing leading '/' from absolute path names
# file: mnt/dist2
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000000000081400

[root@centos-qa-client-2 sbin]# 

[root@centos-qa-client-3 glusterfs]# getfattr -m . -d -e hex /mnt/dist2
getfattr: Removing leading '/' from absolute path names
# file: mnt/dist2
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x000000000007e800

[root@centos-qa-client-3 glusterfs]# 


###############################################################################


[root@centos-qa-client-2 sbin]# ./gluster volume quota dist2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576              1576960
/d2                     1048576                    0
/                       1048576               529408
Comment 3 Saurabh 2011-04-08 10:47:27 UTC 
Inode context is not set on a directory on which a limit is set for a dist-rep volume,





1955                    ret = quota_send_dir_limit_to_cli (frame, this, loc->inode,
(gdb) s
quota_send_dir_limit_to_cli (frame=0x2adb2c2255d0, this=0x431cb20, inode=0x2aaaaff0e15c, name=0x2aaab4003e90 "trusted.limit.list")
    at quota.c:1894
1894            int32_t            ret               = 0;
(gdb) n
1895            char               dir_limit [1024]  = {0, };
(gdb) 
1896            dict_t            *dict              = NULL;
(gdb) 
1897            quota_inode_ctx_t *ctx               = NULL;
(gdb) 
1898            uint64_t           value             = 0;
(gdb) 
1900            ret = inode_ctx_get (inode, this, &value);
(gdb) 
1901            if (ret < 0)
(gdb) 
1924            return ret;
(gdb) 
1925    }


[root@centos-qa-client-2 sbin]# ./gluster volume quota  dist-rep2 list
        path              limit_set          size
----------------------------------------------------------------------------------
/d1                       1.4MB
[root@centos-qa-client-2 sbin]#
Comment 4 Saurabh 2011-04-11 09:46:26 UTC 
[root@centos-qa-client-2 sbin]# ./gluster volume quota dist2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152               180000
/d1                     1048576                80000
/d2                     1048576               100000
/d2/dd2                  102400               100000
/d1/dd1                  102400                80000
[root@centos-qa-client-2 sbin]# 


although /d1/dd1 was also having same amount of data as /d2/dd2, seen for both fuse and nfs.



glusterfs#10.1.12.134:/dist-rep2 on /mnt/gluster-dir type fuse (rw,allow_other,default_permissions,max_read=131072)
10.1.12.134:/dist2 on /mnt/nfs-test type nfs (rw,nfsvers=3,nolock,addr=10.1.12.134)
glusterfs#10.1.12.134:/dist2 on /mnt/gluster-test type fuse (rw,allow_other,default_permissions,max_read=131072)
[root@centos-qa-client-1 dd2]# pwd
/mnt/gluster-test/d2/dd2
[root@centos-qa-client-1 dd2]# ls
f.1  f.10  f.2  f.3  f.4  f.5  f.6  f.7  f.8  f.9
[root@centos-qa-client-1 dd2]# cd ../../d1/dd1
[root@centos-qa-client-1 dd1]# ls
f.1  f.10  f.2  f.3  f.4  f.5  f.6  f.7  f.8  f.9
Comment 5 Raghavendra G 2011-04-13 08:35:00 UTC 
This bug is not consistently reproducible.
Comment 6 Saurabh 2011-04-14 05:23:44 UTC 
In this bug only the last case reported is inconsistently reproducible but all other cases happening were consistent,though after some patches and code correction those are not seen.
Comment 7 Saurabh 2011-04-14 09:43:47 UTC 
Well the problem I mentioned last has occurred again,

[root@centos-qa-client-1 d2]# cd ..
[root@centos-qa-client-1 gluster-test]# ls -R
.:
d1  d2

./d1:
f.1    f.13  f.18  f.22  f.27  f.31  f.36  f.40  f.45  f.5   f.54  f.59  f.63  f.68  f.72  f.77  f.81  f.86  f.90  f.95
f.10   f.14  f.19  f.23  f.28  f.32  f.37  f.41  f.46  f.50  f.55  f.6   f.64  f.69  f.73  f.78  f.82  f.87  f.91  f.96
f.100  f.15  f.2   f.24  f.29  f.33  f.38  f.42  f.47  f.51  f.56  f.60  f.65  f.7   f.74  f.79  f.83  f.88  f.92  f.97
f.11   f.16  f.20  f.25  f.3   f.34  f.39  f.43  f.48  f.52  f.57  f.61  f.66  f.70  f.75  f.8   f.84  f.89  f.93  f.98
f.12   f.17  f.21  f.26  f.30  f.35  f.4   f.44  f.49  f.53  f.58  f.62  f.67  f.71  f.76  f.80  f.85  f.9   f.94  f.99

./d2:
f.1    f.13  f.18  f.22  f.27  f.31  f.36  f.40  f.45  f.5   f.54  f.59  f.63  f.68  f.72  f.77  f.81  f.86  f.90  f.95
f.10   f.14  f.19  f.23  f.28  f.32  f.37  f.41  f.46  f.50  f.55  f.6   f.64  f.69  f.73  f.78  f.82  f.87  f.91  f.96
f.100  f.15  f.2   f.24  f.29  f.33  f.38  f.42  f.47  f.51  f.56  f.60  f.65  f.7   f.74  f.79  f.83  f.88  f.92  f.97
f.11   f.16  f.20  f.25  f.3   f.34  f.39  f.43  f.48  f.52  f.57  f.61  f.66  f.70  f.75  f.8   f.84  f.89  f.93  f.98
f.12   f.17  f.21  f.26  f.30  f.35  f.4   f.44  f.49  f.53  f.58  f.62  f.67  f.71  f.76  f.80  f.85  f.9   f.94  f.99
[root@centos-qa-client-1 gluster-test]# ls d1 | wc -l
100
[root@centos-qa-client-1 gluster-test]# ls -li d1/f.1
27722246 -rw-r--r-- 1 root root 1000 Apr 14 05:25 d1/f.1
[root@centos-qa-client-1 gluster-test]# ls -li d1/f.100
27724117 -rw-r--r-- 1 root root 1000 Apr 14 05:25 d1/f.100
[root@centos-qa-client-1 gluster-test]# ls d2 | wc -l
100

##############################################

[root@centos-qa-client-2 sbin]# ./gluster volume start dr2
Starting volume dr2 has been successful
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152                    0
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152               197000
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------

/                       2097152               198000
[root@centos-qa-client-2 sbin]# 
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152               198000
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152               198000
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr2 | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x000000000000ee48
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr-dr2 | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x000000000000ee48
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr-dr23 | grep size
getfattr: /mnt/dr-dr23: No such file or directory
[root@centos-qa-client-2 sbin]# ls /mnt/dr*
/mnt/dr2:
d1  d2

/mnt/dr23:
d1  d2

/mnt/dr-dr2:
d1  d2
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr23 | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x0000000000014050
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr2/d | grep size
d1/ d2/ 
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr2/d | grep size
d1/ d2/ 
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr2/d1/ | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x0000000000007530
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr23/d1/ | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x000000000000a028
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr23/d2/ | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x000000000000a028
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dr2/d2/ | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x0000000000007918
[root@centos-qa-client-2 sbin]# 


after creating data first time size has shown 197000, and very next attempt 198000 but after that it was intact.

##########################################################################

But again when deleted the above mentioned data and recreated the data, that works fine,
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152               200000
[root@centos-qa-client-2 sbin]#
Comment 8 Anand Avati 2011-04-16 14:51:46 UTC 
PATCH: http://patches.gluster.com/patch/6920 in master (features/marker-quota: fixes in self-healing of dirty inode code path.)
Comment 9 Anand Avati 2011-04-16 14:51:55 UTC 
PATCH: http://patches.gluster.com/patch/6918 in master (marker/quota: use correct byte ordering when updating parent's size.)
Comment 10 Saurabh 2011-04-19 14:45:44 UTC 
bug 764403 is duplicated to this bug and this one is still happening,


[root@centos-qa-client-1 gluster-test]# rm -rf *
[root@centos-qa-client-1 gluster-test]# mkdir d1 d2
[root@centos-qa-client-1 gluster-test]# cd d1
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.586422 seconds, 873 kB/s
[root@centos-qa-client-1 d1]# cd ../d1
[root@centos-qa-client-1 d1]# cd ../d2
[root@centos-qa-client-1 d2]# dd if=/dev/zero of=f.3 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.539047 seconds, 950 kB/s
[root@centos-qa-client-1 d2]# 


[root@centos-qa-client-2 sbin]# ls
gluster  glusterd  glusterfs  glusterfsd
[root@centos-qa-client-2 sbin]# ./gluster volume quota drep disable
Disabling quota has been successful
[root@centos-qa-client-2 sbin]# pwd
/root/quota/inst/sbin
[root@centos-qa-client-2 sbin]# ls
gluster  glusterd  glusterfs  glusterfsd
[root@centos-qa-client-2 sbin]# ./gluster volume quota drep enable
Enabling quota has been successful
[root@centos-qa-client-2 sbin]# ./gluster volume quota drep limit-usage / 2MB
limit set on /
[root@centos-qa-client-2 sbin]# ./gluster volume quota drep list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152                    0
[root@centos-qa-client-2 sbin]#
Comment 11 Saurabh 2011-04-19 14:56:39 UTC 
again for the above mentioned statement for bug764403,

this time I tried to self heal using the command


[root@centos-qa-client-1 gluster-test]# find . | xargs stat


and the size was displayed accordingly,

[root@centos-qa-client-2 sbin]# ./gluster volume quota drep list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152              1040384


But this data is for dist-rep mount over fuse.

Need to check for distributed volume and nfs mount.
Comment 12 Anand Avati 2011-04-21 07:35:54 UTC 
PATCH: http://patches.gluster.com/patch/7010 in master (features/marker-quota: Add the disk space used to parent sizes during creation of new-files or symlinks.)
Comment 13 Anand Avati 2011-04-21 11:38:30 UTC 
PATCH: http://patches.gluster.com/patch/7015 in master (features/marker-quota: handle creation of regular files using mknod.)
Comment 14 Raghavendra G 2011-05-10 06:16:47 UTC 
Hi Saurabh,

Can we close this bug?

regards,
Raghavendra.
Comment 15 Saurabh 2011-05-16 06:48:57 UTC 
Hey Raghu,

This one is also working fine.
Comment 16 Saurabh 2011-06-01 05:41:36 UTC 
Tested the issue for 2671 and found that on fuse mount the issue is not happening for distribute and distribute-replicate volumes.

Tested the same over nfs mounts and its working properly.