| Summary: | Quota: crossing the set limit | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Saurabh <saurabh> |
| Component: | quota | Assignee: | Junaid <junaid> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | mainline | CC: | gluster-bugs, raghavendra, vagarwal |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | --- | |
| Regression: | RTNR | Mount Type: | fuse |
| Documentation: | DNR | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
|
Description
Saurabh
2011-03-29 10:45:17 UTC
Hi, I have a distribute volume with a quota limit set to 2MB, I tried to create dataset in it, and created 2001 files each of 5120bytes, and that is 9MB , hence it crosses the quota limit. BTW, similar issue is seen over nfs mount logs, [root@centos-qa-client-2 sbin]# ./gluster volume info dist1 Volume Name: dist1 Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.1.12.134:/mnt/dist1 Brick2: 10.1.12.135:/mnt/dist1 Options Reconfigured: features.limit-usage: /dist1:2MB features.quota: on ################ from client#################### 10.1.12.134:/dist1 on /mnt/nfs-test type nfs (rw,nfsvers=3,nolock,addr=10.1.12.134) [root@centos-qa-client-1 gluster-test]# cd test [root@centos-qa-client-1 test]# ls -l | wc -l 2001 [root@centos-qa-client-1 test]# ls -li file.199 9240784 -rw-r--r-- 1 root root 5120 Mar 29 06:19 file.199 [root@centos-qa-client-1 test]# bc bc 1.06 Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. 5120*2001 10245120 ./1024 10005 ./1024 9 [root@centos-qa-client-1 test]# pwd /mnt/gluster-test/test Machines involved are, 10.1.12.134/135 as bricks 10.1.12.133 as client login credential is same as "root/syst3m" PATCH: http://patches.gluster.com/patch/6666 in master (features/quota: Use appropriate inodes while logging and create contexts in inode for files.) The issue is already been discussed with the dev guys, new patch is considered for resolving this bug PATCH: http://patches.gluster.com/patch/6683 in master (features/quota: use a double pointer to uint64_t as an argument to dict_get_bin while getting size.) the limit is still crossed, [root@centos-qa-client-1 d1]# bc bc 1.06 Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. 104*10240 1064960 and limit set was just 1 MB. PATCH: http://patches.gluster.com/patch/6764 in master (marker/quota: use correct byte order while storing xattr values in backend.) PATCH: http://patches.gluster.com/patch/6763 in master (cluster/distribute: Account for the first lookup sent to check whether the path is a directory while aggregating quota-xattrs.) PATCH: http://patches.gluster.com/patch/6768 in master (features/quota: use signed type to store delta.) with the latest release this is not happening anymore, checked it over fuse mount the latest happening, is found with dist-strip, test steps, 1. Have a dist-strip volume 2. put some limit on the volume lets say 1MB 3. Create data in it to fill this 1MB. 4. again reduce the limit of the quota , using quota limi-usage command. 5. try to add some file or add some data to an existing file. Expected Result:- it should not allow step 5 to succeed Actual Result: the step 5 is success PATCH: http://patches.gluster.com/patch/6831 in master (features/quota: update quota-limit stored in directory contexts after reconfigure.) The test case that I had mentioned last time in this bug and I tried to verify it on a distribute volume it again failing over nfs mount not over fuse mount. logs, [root@centos-qa-client-2 sbin]# ./gluster volume quota dist4 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1025000 [root@centos-qa-client-2 sbin]# ./gluster volume quota dist4 limit-usage / 500KB limit set on / [root@centos-qa-client-2 sbin]# ./gluster volume quota dist4 list path limit_set size ---------------------------------------------------------------------------------- / 512000 1225005 ################## [root@centos-qa-client-1 g-dist4]# ls -l total 1024 -rw-r--r-- 1 root root 1000 Apr 15 05:28 f.1 -rw-r--r-- 1 root root 512000 Apr 15 05:28 f.2 -rw-r--r-- 1 root root 512000 Apr 15 05:28 f.3 [root@centos-qa-client-1 g-dist4]# cat >> f.1 kkkk cat: write error: Disk quota exceeded [root@centos-qa-client-1 g-dist4]# ls -li total 1024 18481500 -rw-r--r-- 1 root root 1000 Apr 15 05:28 f.1 18481521 -rw-r--r-- 1 root root 512000 Apr 15 05:28 f.2 18481513 -rw-r--r-- 1 root root 512000 Apr 15 05:28 f.3 [root@centos-qa-client-1 g-dist4]# cd ../nfs-dir [root@centos-qa-client-1 nfs-dir]# cat >> f.1 kkkk ll[root@centos-qa-client-1 nfs-dir]# ls -l total 1024 -rw-r--r-- 1 root root 1005 Apr 15 05:31 f.1 -rw-r--r-- 1 root root 512000 Apr 15 05:28 f.2 -rw-r--r-- 1 root root 512000 Apr 15 05:28 f.3 [root@centos-qa-client-1 nfs-dir]# [root@centos-qa-client-1 nfs-dir]# [root@centos-qa-client-1 nfs-dir]# [root@centos-qa-client-1 nfs-dir]# dd if=/dev/zero of=f.4 bs=1KB count=200 200+0 records in 200+0 records out 200000 bytes (200 kB) copied, 0.136956 seconds, 1.5 MB/s [root@centos-qa-client-1 nfs-dir]# ls -l total 1228 -rw-r--r-- 1 root root 1005 Apr 15 05:31 f.1 -rw-r--r-- 1 root root 512000 Apr 15 05:28 f.2 -rw-r--r-- 1 root root 512000 Apr 15 05:28 f.3 -rw-r--r-- 1 root root 200000 Apr 15 05:32 f.4 [root@centos-qa-client-1 nfs-dir]# Well, this is happening only when the limit is reduced, otherwise if without tempering the limit-set for quota is working fine nfs also. After reconfigure, lookup is not sent on the directory with limit set. This is the cause for bug. New issue seen , may be included in this bug, 1. create a fresh dist-rep volume 2. enable quota, set limit 3. mount it using fuse. 4. create two dir's d1 d2. 5. set limit on these dir's 6. create data inside dirs 7. check the cumulative size on the root of the volume 8. delete all of the data from mount point, nfs/fuse. 9. check size again. Expected result:- the size displayed should zero for the root actual result: it is still the same as aggregated earlier. logs, [root@centos-qa-client-1 ~]# cd /mnt/g-drep [root@centos-qa-client-1 g-drep]# ls d1 d2 [root@centos-qa-client-1 g-drep]# rm -rf * [root@centos-qa-client-1 g-drep]# ls -lR .: total 0 [root@centos-qa-client-2 sbin]# ./gluster volume quota drep list path limit_set size ---------------------------------------------------------------------------------- / 3145728 2097152 /d1 1MB /d2 1MB [root@centos-qa-client-1 g-drep]# mkdir d1 d2 [root@centos-qa-client-2 sbin]# ./gluster volume quota drep list path limit_set size ---------------------------------------------------------------------------------- / 3145728 2097152 /d1 1048576 0 /d2 1048576 0 [root@centos-qa-client-1 g-drep]# cd d1 [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.4 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.623266 seconds, 821 kB/s [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.5 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.29657 seconds, 1.7 MB/s [root@centos-qa-client-2 sbin]# #getfattr -m . -d -e hex /mnt/ddrep | grep size [root@centos-qa-client-2 sbin]# ./gluster volume quota drep list path limit_set size ---------------------------------------------------------------------------------- / 3145728 3137536 /d1 1048576 1040384 /d2 1048576 0 [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.6 bs=1KB count=512 dd: writing `f.6': Disk quota exceeded 7+0 records in 6+0 records out 6000 bytes (6.0 kB) copied, 0.004572 seconds, 1.3 MB/s [root@centos-qa-client-2 sbin]# ./gluster volume quota drep list path limit_set size ---------------------------------------------------------------------------------- / 3145728 3145728 /d1 1048576 1048576 /d2 1048576 0 [root@centos-qa-client-1 d1]# ls -li total 1024 18481593 -rw-r--r-- 1 root root 512000 Apr 18 03:55 f.4 18481633 -rw-r--r-- 1 root root 512000 Apr 18 03:55 f.5 18481637 -rw-r--r-- 1 root root 1000 Apr 18 03:55 f.6 [root@centos-qa-client-1 d1]# rm -rf * [root@centos-qa-client-1 d1]# cd .. [root@centos-qa-client-1 g-drep]# rm -rf * [root@centos-qa-client-1 g-drep]# ls [root@centos-qa-client-1 g-drep]# ############################from the server side ################## [root@centos-qa-client-2 sbin]# ./gluster volume quota drep list path limit_set size ---------------------------------------------------------------------------------- / 3145728 2097152 /d1 1MB /d2 1MB [root@centos-qa-client-2 sbin]# over nfs the limit was slightly. This bug is not reproducible on my local setup. rm -rf is working fine bringing down the size to zero. tested it again on the latest release provided by Raghvendra, the last issue reported didn't happen for me also. as discussed, [root@centos-qa-client-3 sbin]# ./gluster volume quota drep list path limit_set size ---------------------------------------------------------------------------------- / 3145728 3145728 /d1 1048576 1048576 /d2 1048576 1081344 /d3 512000 507904 [root@centos-qa-client-3 sbin]# the data created over nfs mount, didn't find the same over fuse mount. Tried to reproduce the last mentioned issue again over nfs mount
[root@centos-qa-client-3 sbin]# ./gluster volume quota drep list
path limit_set size
----------------------------------------------------------------------------------
/ 3145728 1040384
/d1 1048576 1040384
/d2 1048576 0
/d3 512000 0
[root@centos-qa-client-3 sbin]# ./gluster volume quota drep list
path limit_set size
----------------------------------------------------------------------------------
/ 3145728 1056768
/d1 1048576 1056768
/d2 1048576 0
/d3 512000 0
[root@centos-qa-client-3 sbin]#
[root@centos-qa-client-1 nfs-test]# rm -rf *
[root@centos-qa-client-1 nfs-test]# mkdir d1 d2
[root@centos-qa-client-1 nfs-test]# mkdir d3
[root@centos-qa-client-1 nfs-test]# ls
d1 d2 d3
[root@centos-qa-client-1 nfs-test]# cd d1
[root@centos-qa-client-1 d1]# ls
(reverse-i-search)`dd': for i in {1..36}; do dd if=/dev/zero of=f.$i bs=1KB count=10; echo $i; done
[root@centos-qa-client-1 d1]#
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.1 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.623933 seconds, 821 kB/s
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.2 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.265939 seconds, 1.9 MB/s
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1KB count=10
10+0 records in
10+0 records out
10000 bytes (10 kB) copied, 0.106669 seconds, 93.7 kB/s
[root@centos-qa-client-1 d1]#
in connection to the duplicated bug 764403, on dist volumes the data creation before the quota enable/limit set can listed after enabling and setting limit, but there is a glitch, 1. data creation is of about 2MB(approx), limit set is 1MB and the data listed is also 1MB(approx.) 2. increase the limit to 3MB , the actual is then displayed correctly, whereas it should be correct in step 1 itself, Logs, [root@centos-qa-client-1 d2]# cd .. [root@centos-qa-client-1 g-dist2]# ls -lR .: total 32 drwxr-xr-x 2 root root 8192 Apr 20 02:42 d1 drwxr-xr-x 2 root root 8192 Apr 20 02:43 d2 ./d1: total 1016 -rw-r--r-- 1 root root 512000 Apr 20 02:42 f.1 -rw-r--r-- 1 root root 512000 Apr 20 02:42 f.2 ./d2: total 1016 -rw-r--r-- 1 root root 512000 Apr 20 02:43 f.1 -rw-r--r-- 1 root root 512000 Apr 20 02:43 f.2 [root@centos-qa-client-3 sbin]# ./gluster volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 1048576 0 [root@centos-qa-client-1 g-dist2]# find . | xargs stat File: `.' Size: 8192 Blocks: 32 IO Block: 4096 directory Device: 20h/32d Inode: 1 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:46:11.000000000 -0700 Modify: 2011-04-20 02:41:51.000000000 -0700 Change: 2011-04-20 02:41:51.000000000 -0700 File: `./d2' Size: 8192 Blocks: 32 IO Block: 4096 directory Device: 20h/32d Inode: 9240590 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:46:11.000000000 -0700 Modify: 2011-04-20 02:43:36.000000000 -0700 Change: 2011-04-20 02:43:59.000000000 -0700 File: `./d2/f.1' Size: 512000 Blocks: 1016 IO Block: 4096 regular file Device: 20h/32d Inode: 9240593 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:43:31.000000000 -0700 Modify: 2011-04-20 02:43:32.000000000 -0700 Change: 2011-04-20 02:43:32.000000000 -0700 File: `./d2/f.2' Size: 512000 Blocks: 1016 IO Block: 4096 regular file Device: 20h/32d Inode: 9240599 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:43:36.000000000 -0700 Modify: 2011-04-20 02:43:37.000000000 -0700 Change: 2011-04-20 02:43:37.000000000 -0700 File: `./d1' Size: 8192 Blocks: 32 IO Block: 4096 directory Device: 20h/32d Inode: 9240586 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:46:11.000000000 -0700 Modify: 2011-04-20 02:42:33.000000000 -0700 Change: 2011-04-20 02:43:59.000000000 -0700 File: `./d1/f.1' Size: 512000 Blocks: 1016 IO Block: 4096 regular file Device: 20h/32d Inode: 9240592 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:42:21.000000000 -0700 Modify: 2011-04-20 02:42:21.000000000 -0700 Change: 2011-04-20 02:42:21.000000000 -0700 File: `./d1/f.2' Size: 512000 Blocks: 1016 IO Block: 4096 regular file Device: 20h/32d Inode: 9240598 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:42:33.000000000 -0700 Modify: 2011-04-20 02:42:33.000000000 -0700 Change: 2011-04-20 02:42:33.000000000 -0700 [root@centos-qa-client-3 sbin]# ./gluster volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1040384 [root@centos-qa-client-1 g-dist2]# ls d1 d2 [root@centos-qa-client-1 g-dist2]# ls -R .: d1 d2 ./d1: f.1 f.2 ./d2: f.1 f.2 [root@centos-qa-client-1 g-dist2]# ls -lR .: total 32 drwxr-xr-x 2 root root 8192 Apr 20 02:42 d1 drwxr-xr-x 2 root root 8192 Apr 20 02:43 d2 ./d1: total 1016 -rw-r--r-- 1 root root 512000 Apr 20 02:42 f.1 -rw-r--r-- 1 root root 512000 Apr 20 02:42 f.2 ./d2: total 1016 -rw-r--r-- 1 root root 512000 Apr 20 02:43 f.1 -rw-r--r-- 1 root root 512000 Apr 20 02:43 f.2 [root@centos-qa-client-3 sbin]# ./gluster volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 3145728 1040384 [root@centos-qa-client-1 g-dist2]# find . | xargs stat File: `.' Size: 8192 Blocks: 32 IO Block: 4096 directory Device: 20h/32d Inode: 1 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:47:34.000000000 -0700 Modify: 2011-04-20 02:41:51.000000000 -0700 Change: 2011-04-20 02:46:11.000000000 -0700 File: `./d2' Size: 8192 Blocks: 32 IO Block: 4096 directory Device: 20h/32d Inode: 9240590 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:47:34.000000000 -0700 Modify: 2011-04-20 02:43:36.000000000 -0700 Change: 2011-04-20 02:46:11.000000000 -0700 File: `./d2/f.1' Size: 512000 Blocks: 1016 IO Block: 4096 regular file Device: 20h/32d Inode: 9240593 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:43:31.000000000 -0700 Modify: 2011-04-20 02:43:32.000000000 -0700 Change: 2011-04-20 02:43:32.000000000 -0700 File: `./d2/f.2' Size: 512000 Blocks: 1016 IO Block: 4096 regular file Device: 20h/32d Inode: 9240599 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:43:36.000000000 -0700 Modify: 2011-04-20 02:43:37.000000000 -0700 Change: 2011-04-20 02:46:11.000000000 -0700 File: `./d1' Size: 8192 Blocks: 32 IO Block: 4096 directory Device: 20h/32d Inode: 9240586 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:47:34.000000000 -0700 Modify: 2011-04-20 02:42:33.000000000 -0700 Change: 2011-04-20 02:46:07.000000000 -0700 File: `./d1/f.1' Size: 512000 Blocks: 1016 IO Block: 4096 regular file Device: 20h/32d Inode: 9240592 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:42:21.000000000 -0700 Modify: 2011-04-20 02:42:21.000000000 -0700 Change: 2011-04-20 02:42:21.000000000 -0700 File: `./d1/f.2' Size: 512000 Blocks: 1016 IO Block: 4096 regular file Device: 20h/32d Inode: 9240598 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-04-20 02:42:33.000000000 -0700 Modify: 2011-04-20 02:42:33.000000000 -0700 Change: 2011-04-20 02:46:07.000000000 -0700 [root@centos-qa-client-1 g-dist2]# [root@centos-qa-client-3 sbin]# ./gluster volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 3145728 2080768 [root@centos-qa-client-3 sbin]# Hi Saurabh, The bug is not reproducible on my laptop. Can you try it once again? Most likely the issue will be lookups not being sent to marker on server-side (this may be due to caching of inodes by kernel). What you can do is mount the volume with entry-timeout=0 and do 'find . | xargs stat > /dev/null' and then trying your tests. You can use the scripts I had sent you earlier to confirm whether xattrs are set properly. regards, Raghavendra. Hello Raghu, This is not for me on the 3.2.0 release, I tested it again. so we can close this bug. tested it earlier itself |