Hide Forgot
Filing this bug other than bug764381, as this new issue may be related to stripe also. from client end, [root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=512 dd: closing output file `f.1': Disk quota exceeded [root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=500 500+0 records in 500+0 records out 500000 bytes (500 kB) copied, 0.249811 seconds, 2.0 MB/s [root@centos-qa-client-1 dd1]# ls -li total 136 4621124 -rw-r--r-- 1 root root 500000 Apr 14 06:11 f.1 [root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.2 bs=1KB count=1 dd: opening `f.2': Disk quota exceeded [root@centos-qa-client-1 dd1]# ls -li total 136 4621124 -rw-r--r-- 1 root root 500000 Apr 14 06:11 f.1 [root@centos-qa-client-1 dd1]# mount | grep ds2 glusterfs#10.1.12.134:/ds2 on /mnt/g-ds2 type fuse (rw,allow_other,default_permissions,max_read=131072) [root@centos-qa-client-1 dd1]# #################From server side################# [root@centos-qa-client-2 sbin]# ./gluster volume quota ds2 list path limit_set size ---------------------------------------------------------------------------------- / 2097152 131072 /d1 1048576 131072 /d2 1048576 0 /d1/dd1 512000 131072 /d2/dd2 512000 0 #################vol info and xattr###################### [root@centos-qa-client-2 sbin]# ./gluster volume info ds2 Volume Name: ds2 Type: Stripe Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: 10.1.12.134:/mnt/ds2 Brick2: 10.1.12.134:/mnt/ds-ds2 Brick3: 10.1.12.135:/mnt/ds2 Brick4: 10.1.12.135:/mnt/ds-ds2 [root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/ds2 | grep size getfattr: Removing leading '/' from absolute path names trusted.glusterfs.quota.size=0x0000000000020000 [root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/ds2/d1 | grep size getfattr: Removing leading '/' from absolute path names trusted.glusterfs.quota.size=0x0000000000020000 [root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/ds-ds2/ | grep size getfattr: Removing leading '/' from absolute path names trusted.glusterfs.quota.size=0x0000000000040000 from other server [root@centos-qa-client-3 sbin]# getfattr -m . -d -e hex /mnt/ds2/d1 | grep size getfattr: Removing leading '/' from absolute path names trusted.glusterfs.quota.size=0x0000000000060000 [root@centos-qa-client-3 sbin]# getfattr -m . -d -e hex /mnt/ds-ds2/d1 | grep size getfattr: Removing leading '/' from absolute path names trusted.glusterfs.quota.size=0x000000000007a120
PATCH: http://patches.gluster.com/patch/6901 in master (features/marker-quota: Use the actual disk-space consumed instead of ia_size returned in iatt for calculation of sizes.)
PATCH: http://patches.gluster.com/patch/6902 in master (features/quota: use disk usage instead of file-sizes.)
PATCH: http://patches.gluster.com/patch/6903 in master (cluster/stripe: aggregate xattrs holding quota-size in lookup and getxattr calls.)
[root@centos-qa-client-3 ~]# cd quota/inst/sbin/;./gluster volume quota ds2 list path limit_set size ---------------------------------------------------------------------------------- / 2097152 2641920 [root@centos-qa-client-3 sbin]# cd quota/inst/sbin/;./gluster volume quota ds2 list -bash: cd: quota/inst/sbin/: No such file or directory path limit_set size ---------------------------------------------------------------------------------- / 2097152 3170304 [root@centos-qa-client-3 sbin]# ./gluster volume info ds2 Volume Name: ds2 Type: Distributed-Stripe Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.1.12.134:/mnt/ds2 Brick2: 10.1.12.134:/mnt/dds2 Brick3: 10.1.12.135:/mnt/ds2 Brick4: 10.1.12.135:/mnt/dds2 Options Reconfigured: features.limit-usage: /:2MB features.quota: on [root@centos-qa-client-3 sbin]# #################################################3 [root@centos-qa-client-1 gluster-test]# ls -lR .: total 560 drwxr-xr-x 2 root root 8192 Apr 19 02:21 d -rw-r--r-- 1 root root 512000 Apr 19 01:47 f.1 -rw-r--r-- 1 root root 512000 Apr 19 03:11 f.3 ./d: total 1056 -rw-r--r-- 1 root root 512000 Apr 19 02:20 f.1 -rw-r--r-- 1 root root 512000 Apr 19 02:21 f.2 -rw-r--r-- 1 root root 512000 Apr 19 02:21 f.3 -rw-r--r-- 1 root root 512000 Apr 19 02:21 f.4 [root@centos-qa-client-1 gluster-test]# outgrows the set limit, also per file the size differs from the mount point. [root@centos-qa-client-3 sbin]# ls -l /mnt/ds2 total 8 drwxr-xr-x 2 root root 4096 Apr 19 02:21 d [root@centos-qa-client-3 sbin]# ls -l /mnt/ds2/d total 792 -rw-r--r-- 1 root root 393216 Apr 19 02:20 f.1 -rw-r--r-- 1 root root 393216 Apr 19 02:21 f.2 -rw-r--r-- 1 root root 393216 Apr 19 02:21 f.3 [root@centos-qa-client-3 sbin]# ls -l /mnt/dds2/d total 756 -rw-r--r-- 1 root root 512000 Apr 19 02:20 f.1 -rw-r--r-- 1 root root 512000 Apr 19 02:21 f.2 -rw-r--r-- 1 root root 512000 Apr 19 02:21 f.3 [root@centos-qa-client-3 sbin]#
PATCH: http://patches.gluster.com/patch/7197 in master (cluster/stripe: Fix a typo in strncmp.)
PATCH: http://patches.gluster.com/patch/7348 in release-3.2 (cluster/stripe: Fix a typo in strncmp.)
[root@centos-qa-client-2 ~]# /usr/sbin/gluster volume info dstripe Volume Name: dstripe Type: Distributed-Stripe Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.1.12.134:/mnt/dstripe Brick2: 10.1.12.135:/mnt/dstripe Brick3: 10.1.12.134:/mnt/d-dstripe Brick4: 10.1.12.135:/mnt/d-dstripe Options Reconfigured: features.limit-usage: /:1MB features.quota: on [root@centos-qa-client-2 ~]# /usr/sbin/gluster volume quota dstripe list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1056768 [root@centos-qa-client-2 ~]# ###############from client###################### [root@centos-qa-3 gluster-test]# stat f.3 File: `f.3' Size: 512000 Blocks: 1032 IO Block: 4096 regular file Device: 18h/24d Inode: 7077922 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 1969-12-31 16:00:00.000000000 -0800 Modify: 2011-06-01 01:16:49.000000000 -0700 Change: 2011-06-01 01:16:52.000000000 -0700 [root@centos-qa-3 gluster-test]# bc bc 1.06 Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. 1032*512 528384 [root@centos-qa-3 gluster-test]# stat 1 File: `1' Size: 512000 Blocks: 1032 IO Block: 4096 regular file Device: 18h/24d Inode: 7077918 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 1969-12-31 16:00:00.000000000 -0800 Modify: 2011-06-01 01:09:40.000000000 -0700 Change: 2011-06-01 01:16:52.000000000 -0700 [root@centos-qa-3 gluster-test]# stat 2 File: `2' Size: 512000 Blocks: 1032 IO Block: 4096 regular file Device: 18h/24d Inode: 7077921 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 1969-12-31 16:00:00.000000000 -0800 Modify: 2011-06-01 01:09:27.000000000 -0700 Change: 2011-06-01 01:16:52.000000000 -0700 [root@centos-qa-3 gluster-test]# bc bc 1.06 Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. 528384*3 1585152 [root@centos-qa-3 gluster-test]# ls -l total 792 -rw-r--r-- 1 root root 512000 Jun 1 2011 1 -rw-r--r-- 1 root root 512000 Jun 1 2011 2 -rw-r--r-- 1 root root 512000 Jun 1 2011 f.3 [root@centos-qa-3 gluster-test]# #####################################################3 [root@centos-qa-3 gluster-test]# touch f.4 touch: cannot touch `f.4': No such file or directory [root@centos-qa-3 gluster-test]# ls -l total 792 -rw-r--r-- 1 root root 512000 Jun 1 2011 1 -rw-r--r-- 1 root root 512000 Jun 1 2011 2 -rw-r--r-- 1 root root 512000 Jun 1 2011 f.3 [root@centos-qa-3 gluster-test]# vim f.4 [root@centos-qa-3 gluster-test]# ls -l total 800 -rw-r--r-- 1 root root 512000 Jun 1 2011 1 -rw-r--r-- 1 root root 512000 Jun 1 2011 2 -rw-r--r-- 1 root root 512000 Jun 1 2011 f.3 -rw-r--r-- 1 root root 8 Jun 1 2011 f.4 [root@centos-qa-3 gluster-test]# dd if=/dev/zero of=f.4 bs=512000 count=1 dd: closing output file `f.4': Disk quota exceeded [root@centos-qa-3 gluster-test]# ls -li total 928 7077918 -rw-r--r-- 1 root root 512000 Jun 1 2011 1 7077921 -rw-r--r-- 1 root root 512000 Jun 1 2011 2 7077922 -rw-r--r-- 1 root root 512000 Jun 1 2011 f.3 7077925 -rw-r--r-- 1 root root 131072 Jun 1 2011 f.4 [root@centos-qa-3 gluster-test]# ############# from server end ################### [root@centos-qa-client-2 ~]# /usr/sbin/gluster volume quota dstripe list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1728512 [root@centos-qa-client-2 ~]#
gluster> volume info str-marker Volume Name: str-marker Type: Distributed-Stripe Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: junaid-laptop:/export/str1 Brick2: junaid-laptop:/export/str2 Brick3: junaid-laptop:/export/str3 Brick4: junaid-laptop:/export/str4 Options Reconfigured: diagnostics.brick-log-level: NONE features.quota: on features.limit-usage: /:100MB ################################################# gluster> volume quota str-marker list path limit_set size ---------------------------------------------------------------------------------- / 104857600 0 ################################################# root@junaid-laptop:/mount# du -h /tmp/file1 3.2M /tmp/file1 root@junaid-laptop:/mount# cp /tmp/file1 file gluster> volume quota str-marker list path limit_set size ---------------------------------------------------------------------------------- / 104857600 3289088 ################################################# root@junaid-laptop:/mount# rm file gluster> volume quota str-marker list path limit_set size ---------------------------------------------------------------------------------- / 104857600 0 ################################################# root@junaid-laptop:/mount# for i in {1..10}; do cp /tmp/file1 file.$i; done gluster> volume quota str-marker list path limit_set size ---------------------------------------------------------------------------------- / 104857600 32890880 ################################################## root@junaid-laptop:/mount# rm -rf * root@junaid-laptop:/mount# dd if=/dev/zero of=file bs=1KB count=100 gluster> volume quota str-marker list path limit_set size ---------------------------------------------------------------------------------- / 104857600 106496 root@junaid-laptop:/mount# ls -lR gluster> volume quota str-marker list path limit_set size ---------------------------------------------------------------------------------- / 104857600 110592 #################################################### So, what I am trying to convey is that when we are writing small files, the last write is not being received by the second stripe server which I confirmed by attaching gdb to the second stripe server. So, on next lookup the quota updation updates the xattr size. But in case of large file it works fine (I think even there one of the blocs is left out, but since it doesn't affect the overall output much). I also changed 512*buf.ia_blocks to buf.ia_size in the code, and it works fine for small and large files. The below output is got after making the above said change. gluster> volume quota str-marker list path limit_set size ---------------------------------------------------------------------------------- / 104857600 0 root@junaid-laptop:/mount# for i in {1..10}; do dd if=/dev/zero of=file.$i bs=1KB count=100; done gluster> volume quota str-marker list path limit_set size ---------------------------------------------------------------------------------- / 104857600 1000000
what is the case of handling the quota limit on sparse files? If we consider buf->ia_size, then we may miss out on the holes, which is mostly the case with VMs.
Currently, Quota doesn't treat sparse files as special files. It handles it by calculating the size using buf->ia_blocks x 512.
(In reply to comment #13) > Currently, Quota doesn't treat sparse files as special files. It handles it by > calculating the size using buf->ia_blocks x 512. Exactly that is the point. We are calculating based on 'ia_blocks' and if we display using 'ia_size' there will be surely mismatches. Try it for a example file created like this: bash# dd if=/dev/urandam of=/mount/point/of/stripe-volume/file seek=20000 bs=4k count=1
The value is shown correctly on the next access to the file.