Bug 764492 (GLUSTER-2760)

Summary: Quota: stripe volume not showing the quota size properly
Product: [Community] GlusterFS Reporter: Saurabh <saurabh>
Component: quotaAssignee: Junaid <junaid>
Status: CLOSED WORKSFORME QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: mainlineCC: amarts, gluster-bugs, sgowda, vagarwal
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Saurabh 2011-04-14 13:33:00 UTC
Filing this bug other than bug764381, as this new issue may be related to stripe also.

  from client end,

[root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=512
dd: closing output file `f.1': Disk quota exceeded
[root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=500
500+0 records in
500+0 records out
500000 bytes (500 kB) copied, 0.249811 seconds, 2.0 MB/s
[root@centos-qa-client-1 dd1]# ls -li 
total 136
4621124 -rw-r--r-- 1 root root 500000 Apr 14 06:11 f.1
[root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.2 bs=1KB count=1
dd: opening `f.2': Disk quota exceeded
[root@centos-qa-client-1 dd1]# ls -li 
total 136
4621124 -rw-r--r-- 1 root root 500000 Apr 14 06:11 f.1
[root@centos-qa-client-1 dd1]# mount | grep ds2
glusterfs#10.1.12.134:/ds2 on /mnt/g-ds2 type fuse (rw,allow_other,default_permissions,max_read=131072)
[root@centos-qa-client-1 dd1]# 


#################From server side#################
[root@centos-qa-client-2 sbin]# ./gluster volume quota ds2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152               131072
/d1                     1048576               131072
/d2                     1048576                    0
/d1/dd1                  512000               131072
/d2/dd2                  512000                    0

#################vol info and xattr######################
[root@centos-qa-client-2 sbin]# ./gluster volume info ds2

Volume Name: ds2
Type: Stripe
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 10.1.12.134:/mnt/ds2
Brick2: 10.1.12.134:/mnt/ds-ds2
Brick3: 10.1.12.135:/mnt/ds2
Brick4: 10.1.12.135:/mnt/ds-ds2


[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/ds2 | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x0000000000020000

[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/ds2/d1 | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x0000000000020000
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/ds-ds2/ | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x0000000000040000


from other server

[root@centos-qa-client-3 sbin]# getfattr -m . -d -e hex /mnt/ds2/d1 | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x0000000000060000
[root@centos-qa-client-3 sbin]# getfattr -m . -d -e hex /mnt/ds-ds2/d1 | grep size
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.size=0x000000000007a120

Comment 1 Anand Avati 2011-04-16 03:02:56 UTC
PATCH: http://patches.gluster.com/patch/6901 in master (features/marker-quota: Use the actual disk-space consumed instead of ia_size returned in iatt for calculation of sizes.)

Comment 2 Anand Avati 2011-04-16 03:03:01 UTC
PATCH: http://patches.gluster.com/patch/6902 in master (features/quota: use disk usage instead of file-sizes.)

Comment 3 Anand Avati 2011-04-16 03:03:05 UTC
PATCH: http://patches.gluster.com/patch/6903 in master (cluster/stripe: aggregate xattrs holding quota-size in lookup and getxattr calls.)

Comment 4 Anand Avati 2011-04-16 03:06:33 UTC
PATCH: http://patches.gluster.com/patch/6901 in master (features/marker-quota: Use the actual disk-space consumed instead of ia_size returned in iatt for calculation of sizes.)

Comment 5 Anand Avati 2011-04-16 03:06:37 UTC
PATCH: http://patches.gluster.com/patch/6902 in master (features/quota: use disk usage instead of file-sizes.)

Comment 6 Anand Avati 2011-04-16 03:06:43 UTC
PATCH: http://patches.gluster.com/patch/6903 in master (cluster/stripe: aggregate xattrs holding quota-size in lookup and getxattr calls.)

Comment 7 Saurabh 2011-04-19 07:36:50 UTC
[root@centos-qa-client-3 ~]# cd quota/inst/sbin/;./gluster volume quota ds2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152              2641920
[root@centos-qa-client-3 sbin]# cd quota/inst/sbin/;./gluster volume quota ds2 list
-bash: cd: quota/inst/sbin/: No such file or directory
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       2097152              3170304
[root@centos-qa-client-3 sbin]# ./gluster volume info ds2

Volume Name: ds2
Type: Distributed-Stripe
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.1.12.134:/mnt/ds2
Brick2: 10.1.12.134:/mnt/dds2
Brick3: 10.1.12.135:/mnt/ds2
Brick4: 10.1.12.135:/mnt/dds2
Options Reconfigured:
features.limit-usage: /:2MB
features.quota: on
[root@centos-qa-client-3 sbin]# 



#################################################3

[root@centos-qa-client-1 gluster-test]# ls -lR
.:
total 560
drwxr-xr-x 2 root root   8192 Apr 19 02:21 d
-rw-r--r-- 1 root root 512000 Apr 19 01:47 f.1
-rw-r--r-- 1 root root 512000 Apr 19 03:11 f.3

./d:
total 1056
-rw-r--r-- 1 root root 512000 Apr 19 02:20 f.1
-rw-r--r-- 1 root root 512000 Apr 19 02:21 f.2
-rw-r--r-- 1 root root 512000 Apr 19 02:21 f.3
-rw-r--r-- 1 root root 512000 Apr 19 02:21 f.4
[root@centos-qa-client-1 gluster-test]# 


outgrows the set limit, also per file the size differs from the mount point.

[root@centos-qa-client-3 sbin]# ls -l /mnt/ds2
total 8
drwxr-xr-x 2 root root 4096 Apr 19 02:21 d
[root@centos-qa-client-3 sbin]# ls -l /mnt/ds2/d
total 792
-rw-r--r-- 1 root root 393216 Apr 19 02:20 f.1
-rw-r--r-- 1 root root 393216 Apr 19 02:21 f.2
-rw-r--r-- 1 root root 393216 Apr 19 02:21 f.3
[root@centos-qa-client-3 sbin]# ls -l /mnt/dds2/d
total 756
-rw-r--r-- 1 root root 512000 Apr 19 02:20 f.1
-rw-r--r-- 1 root root 512000 Apr 19 02:21 f.2
-rw-r--r-- 1 root root 512000 Apr 19 02:21 f.3
[root@centos-qa-client-3 sbin]#

Comment 8 Anand Avati 2011-05-30 05:49:40 UTC
PATCH: http://patches.gluster.com/patch/7197 in master (cluster/stripe: Fix a typo in strncmp.)

Comment 9 Anand Avati 2011-06-01 04:30:08 UTC
PATCH: http://patches.gluster.com/patch/7348 in release-3.2 (cluster/stripe: Fix a typo in strncmp.)

Comment 10 Saurabh 2011-06-01 05:25:19 UTC
[root@centos-qa-client-2 ~]# /usr/sbin/gluster volume info dstripe

Volume Name: dstripe
Type: Distributed-Stripe
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.1.12.134:/mnt/dstripe
Brick2: 10.1.12.135:/mnt/dstripe
Brick3: 10.1.12.134:/mnt/d-dstripe
Brick4: 10.1.12.135:/mnt/d-dstripe
Options Reconfigured:
features.limit-usage: /:1MB
features.quota: on



[root@centos-qa-client-2 ~]# /usr/sbin/gluster volume quota dstripe list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576              1056768
[root@centos-qa-client-2 ~]# 


###############from client######################

[root@centos-qa-3 gluster-test]# stat f.3
  File: `f.3'
  Size: 512000    	Blocks: 1032       IO Block: 4096   regular file
Device: 18h/24d	Inode: 7077922     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 1969-12-31 16:00:00.000000000 -0800
Modify: 2011-06-01 01:16:49.000000000 -0700
Change: 2011-06-01 01:16:52.000000000 -0700
[root@centos-qa-3 gluster-test]# bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
1032*512
528384
[root@centos-qa-3 gluster-test]# stat 1
  File: `1'
  Size: 512000    	Blocks: 1032       IO Block: 4096   regular file
Device: 18h/24d	Inode: 7077918     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 1969-12-31 16:00:00.000000000 -0800
Modify: 2011-06-01 01:09:40.000000000 -0700
Change: 2011-06-01 01:16:52.000000000 -0700
[root@centos-qa-3 gluster-test]# stat 2
  File: `2'
  Size: 512000    	Blocks: 1032       IO Block: 4096   regular file
Device: 18h/24d	Inode: 7077921     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 1969-12-31 16:00:00.000000000 -0800
Modify: 2011-06-01 01:09:27.000000000 -0700
Change: 2011-06-01 01:16:52.000000000 -0700
[root@centos-qa-3 gluster-test]# bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
528384*3
1585152
[root@centos-qa-3 gluster-test]# ls -l
total 792
-rw-r--r-- 1 root root 512000 Jun  1  2011 1
-rw-r--r-- 1 root root 512000 Jun  1  2011 2
-rw-r--r-- 1 root root 512000 Jun  1  2011 f.3
[root@centos-qa-3 gluster-test]# 

#####################################################3

[root@centos-qa-3 gluster-test]# touch f.4
touch: cannot touch `f.4': No such file or directory
[root@centos-qa-3 gluster-test]# ls -l
total 792
-rw-r--r-- 1 root root 512000 Jun  1  2011 1
-rw-r--r-- 1 root root 512000 Jun  1  2011 2
-rw-r--r-- 1 root root 512000 Jun  1  2011 f.3
[root@centos-qa-3 gluster-test]# vim f.4
[root@centos-qa-3 gluster-test]# ls -l
total 800
-rw-r--r-- 1 root root 512000 Jun  1  2011 1
-rw-r--r-- 1 root root 512000 Jun  1  2011 2
-rw-r--r-- 1 root root 512000 Jun  1  2011 f.3
-rw-r--r-- 1 root root      8 Jun  1  2011 f.4
[root@centos-qa-3 gluster-test]# dd if=/dev/zero of=f.4 bs=512000 count=1
dd: closing output file `f.4': Disk quota exceeded
[root@centos-qa-3 gluster-test]# ls -li
total 928
7077918 -rw-r--r-- 1 root root 512000 Jun  1  2011 1
7077921 -rw-r--r-- 1 root root 512000 Jun  1  2011 2
7077922 -rw-r--r-- 1 root root 512000 Jun  1  2011 f.3
7077925 -rw-r--r-- 1 root root 131072 Jun  1  2011 f.4
[root@centos-qa-3 gluster-test]# 

############# from server end ###################

[root@centos-qa-client-2 ~]# /usr/sbin/gluster volume quota dstripe list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576              1728512
[root@centos-qa-client-2 ~]#

Comment 11 Junaid 2011-06-03 04:51:44 UTC
gluster> volume info str-marker
 
Volume Name: str-marker
Type: Distributed-Stripe
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: junaid-laptop:/export/str1
Brick2: junaid-laptop:/export/str2
Brick3: junaid-laptop:/export/str3
Brick4: junaid-laptop:/export/str4
Options Reconfigured:
diagnostics.brick-log-level: NONE
features.quota: on
features.limit-usage: /:100MB
#################################################

gluster> volume quota str-marker list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                     104857600                    0
#################################################

root@junaid-laptop:/mount# du -h /tmp/file1 
3.2M	/tmp/file1

root@junaid-laptop:/mount# cp /tmp/file1 file

gluster> volume quota str-marker list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                     104857600              3289088
#################################################

root@junaid-laptop:/mount# rm file 

gluster> volume quota str-marker list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                     104857600                    0
#################################################

root@junaid-laptop:/mount# for i in {1..10}; do cp /tmp/file1 file.$i; done


gluster> volume quota str-marker list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                     104857600             32890880
##################################################

root@junaid-laptop:/mount# rm -rf *
root@junaid-laptop:/mount# dd if=/dev/zero of=file bs=1KB count=100
gluster> volume quota str-marker list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                     104857600               106496

root@junaid-laptop:/mount# ls -lR

gluster> volume quota str-marker list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                     104857600               110592
####################################################

So, what I am trying to convey is that when we are writing small files, the last write is not being received by the second stripe server which I confirmed by attaching gdb to the second stripe server. So, on next lookup the quota updation updates the xattr size. But in case of large file it works fine (I think even there one of the blocs is left out, but since it doesn't affect the overall output much).

I also changed 512*buf.ia_blocks to buf.ia_size in the code, and it works fine for small and large files. The below output is got after making the above said change.
gluster> volume quota str-marker list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                     104857600                    0

root@junaid-laptop:/mount# for i in {1..10}; do dd if=/dev/zero of=file.$i bs=1KB count=100; done


gluster> volume quota str-marker list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                     104857600              1000000

Comment 12 Amar Tumballi 2011-06-03 05:09:25 UTC
what is the case of handling the quota limit on sparse files? If we consider buf->ia_size, then we may miss out on the holes, which is mostly the case with VMs.

Comment 13 Junaid 2011-06-03 05:35:23 UTC
Currently, Quota doesn't treat sparse files as special files. It handles it by calculating the size using buf->ia_blocks x 512.

Comment 14 Amar Tumballi 2011-06-03 05:39:28 UTC
(In reply to comment #13)
> Currently, Quota doesn't treat sparse files as special files. It handles it by
> calculating the size using buf->ia_blocks x 512.

Exactly that is the point. We are calculating based on 'ia_blocks' and if we display using 'ia_size' there will be surely mismatches.

Try it for a example file created like this:

bash# dd if=/dev/urandam of=/mount/point/of/stripe-volume/file seek=20000 bs=4k count=1

Comment 15 Junaid 2011-06-09 04:33:32 UTC
The value is shown correctly on the next access to the file.