Hide Forgot
Same issue happened in this scenario also, [root@centos-qa-client-1 gluster-test]# ls -R .: d1 d2 ./d1: dd1 ./d1/dd1: ./d2: dd2 ./d2/dd2: [root@centos-qa-client-1 gluster-test]# cd d1/dd1 [root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=512 dd: closing output file `f.1': Disk quota exceeded [root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=500 dd: writing `f.1': Disk quota exceeded dd: closing output file `f.1': Disk quota exceeded [root@centos-qa-client-1 dd1]# ls -li total 4 18481504 -rw-r--r-- 1 root root 0 Apr 13 02:31 f.1 [root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=200 dd: writing `f.1': Disk quota exceeded dd: closing output file `f.1': Disk quota exceeded [root@centos-qa-client-1 dd1]# rm -rf * [root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.2 bs=1KB count=400 dd: opening `f.2': Disk quota exceeded [root@centos-qa-client-1 dd1]# ls -lia total 64 18481468 drwxr-xr-x 2 root root 16384 Apr 13 02:31 . 18481440 drwxr-xr-x 3 root root 16384 Apr 13 02:29 .. [root@centos-qa-client-1 dd1]# #################################### [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-rep2 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 0 /d1/dd1 512000 0 /d2 1048576 0 /d2/dd2 512000 0 [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-rep2 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 0 /d1/dd1 512000 0 /d2 1048576 0 /d2/dd2 512000 0 [root@centos-qa-client-2 sbin]#
Setup, Vol: Dist-strip, Quota: on Quota limit set: on a diretory and the root not having any limit set. test, 1. create a dir over fuse mount. 2. set the quota limit to 1MB. 3. create a file using "dd if=/dev/zero of=f.1 bs=1KB count=512" 4. create one more file using same command. 5. Now use the same dd command as in step4 for creating the same file again. 6. try creating some more files, Expected Result:- the steps 5,6 should be success Actual Result:- The steps 5,6 are failure. Logs, From fuse client, [root@centos-qa-client-1 g-diststrip]# cd d1 [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.2 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.209521 seconds, 2.4 MB/s [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.2 bs=1KB count=100 dd: writing `f.2': Disk quota exceeded dd: closing output file `f.2': Disk quota exceeded [root@centos-qa-client-1 d1]# ls -li total 512 9240706 -rw-r--r-- 1 root root 512000 Apr 13 00:03 f.1 9240604 -rw-r--r-- 1 root root 0 Apr 13 01:06 f.2 [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.2 bs=1KB count=512 dd: writing `f.2': Disk quota exceeded dd: closing output file `f.2': Disk quota exceeded [root@centos-qa-client-1 d1]# rm f.2 rm: remove regular empty file `f.2'? n [root@centos-qa-client-1 d1]# ls -li total 512 9240706 -rw-r--r-- 1 root root 512000 Apr 13 00:03 f.1 9240604 -rw-r--r-- 1 root root 0 Apr 13 01:07 f.2 [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1KB count=512 dd: opening `f.3': Disk quota exceeded [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1KB count=1 dd: opening `f.3': Disk quota exceeded [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1 count=1 dd: opening `f.3': Disk quota exceeded [root@centos-qa-client-1 d1]# ls -li total 512 9240706 -rw-r--r-- 1 root root 512000 Apr 13 00:03 f.1 9240604 -rw-r--r-- 1 root root 0 Apr 13 01:07 f.2 [root@centos-qa-client-1 d1]# rm f.2 rm: remove regular empty file `f.2'? y [root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1 count=1 dd: opening `f.3': Disk quota exceeded [root@centos-qa-client-1 d1]# ls -li total 508 9240706 -rw-r--r-- 1 root root 512000 Apr 13 00:03 f.1 [root@centos-qa-client-1 d1]# pwd /mnt/g-diststrip/d1 [root@centos-qa-client-1 d1]# mount | strip1 -bash: strip1: command not found [root@centos-qa-client-1 d1]# mount | grep strip1 glusterfs#10.1.12.134:/dist-strip1 on /mnt/g-diststrip type fuse (rw,allow_other,default_permissions,max_read=131072) [root@centos-qa-client-1 d1]# [root@centos-qa-client-1 d1]# #########################################################3 from server end, [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 512000 [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 1024000 [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 512000 [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 512000 [root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dist-strip1/d1/f.1 getfattr: Removing leading '/' from absolute path names # file: mnt/dist-strip1/d1/f.1 trusted.gfid=0xc44457bfab5744fc8e04b5c6ba0459e2 trusted.glusterfs.quota.11126cff-4e45-47d3-93ba-0cefb5b97a11.contri=0x000000000007d000 [root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dist-strip1/d1/f.2 getfattr: Removing leading '/' from absolute path names # file: mnt/dist-strip1/d1/f.2 trusted.gfid=0xdefa4d77d89047238d85558fdedc9d92 trusted.glusterfs.quota.11126cff-4e45-47d3-93ba-0cefb5b97a11.contri=0x0000000000000000 [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 512000 [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 512000 [root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dist-strip1/ getfattr: Removing leading '/' from absolute path names # file: mnt/dist-strip1 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000007ffffffe trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x000000000007d000 trusted.glusterfs.test=0x776f726b696e6700 [root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 512000 [root@centos-qa-client-2 sbin]#
PATCH: http://patches.gluster.com/patch/6864 in master (quota/(f)truncate: pass correct delta for updating parent sizes after successful completion.)
I am having qa-12 release for glusterfs and this issue again happening in this kind of envrionment, Well, the volume is a striped volume. [root@centos-qa-client-1 g-ds2]# mkdir d3 [root@centos-qa-client-1 g-ds2]# mkdir d3/dd3 [root@centos-qa-client-1 g-ds2]# cd d3/dd3 [root@centos-qa-client-1 dd3]# ls [root@centos-qa-client-1 dd3]# dd if=/dev/zero of=f.1 bs=1KB count=500 500+0 records in 500+0 records out 500000 bytes (500 kB) copied, 0.170084 seconds, 2.9 MB/s [root@centos-qa-client-1 dd3]# cd .. [root@centos-qa-client-1 d3]# rm -rf dd3 [root@centos-qa-client-1 d3]# dd if=/dev/zero of=f.1 bs=1KB count=500 dd: closing output file `f.1': Disk quota exceeded [root@centos-qa-client-1 d3]# ls -li total 136 4621122 -rw-r--r-- 1 root root 458000 Apr 14 06:50 f.1 [root@centos-qa-client-1 d3]# cd .. [root@centos-qa-client-1 g-ds2]# mkdir d4 [root@centos-qa-client-1 g-ds2]# mkdir d4/dd4 [root@centos-qa-client-1 g-ds2]# cd d4/dd4 [root@centos-qa-client-1 dd4]# dd if=/dev/zero of=f.1 bs=1KB count=500 500+0 records in 500+0 records out 500000 bytes (500 kB) copied, 0.168528 seconds, 3.0 MB/s [root@centos-qa-client-1 dd4]# cd .. [root@centos-qa-client-1 d4]# rm -rf dd4 [root@centos-qa-client-1 d4]# dd if=/dev/zero of=f.1 bs=1KB count=500 500+0 records in 500+0 records out 500000 bytes (500 kB) copied, 0.168964 seconds, 3.0 MB/s [root@centos-qa-client-1 d4]# ls -li total 136 4621128 -rw-r--r-- 1 root root 500000 Apr 14 07:01 f.1 [root@centos-qa-client-1 d4]# dd if=/dev/zero of=f.2 bs=1KB count=500 dd: opening `f.2': Disk quota exceeded [root@centos-qa-client-1 d4]# ls -li total 136 4621128 -rw-r--r-- 1 root root 500000 Apr 14 07:01 f.1 [root@centos-qa-client-1 d4]# dd if=/dev/zero of=f.2 bs=1KB count=300 dd: opening `f.2': Disk quota exceeded [root@centos-qa-client-1 d4]# ls -li total 136 4621128 -rw-r--r-- 1 root root 500000 Apr 14 07:01 f.1 [root@centos-qa-client-1 d4]# dd if=/dev/zero of=f.2 bs=1KB count=1 dd: opening `f.2': Disk quota exceeded [root@centos-qa-client-1 d4]# ls -li total 136 4621128 -rw-r--r-- 1 root root 500000 Apr 14 07:01 f.1 [root@centos-qa-client-1 d4]# ####################################### [root@centos-qa-client-2 sbin]# ./gluster volume quota ds2 list path limit_set size ---------------------------------------------------------------------------------- /d1 1048576 131072 /d1/dd1 512000 131072 /d2/dd2 500KB /d3 1048576 131072 /d3/dd3 500KB /d4 1048576 131072 /d4/dd4 500KB The size are not shown correctly because of bug764492,
Saurabh, Striped volumes have some known issues with quota (as marker works on st_size instead of st_blocks). Striped volumes will consume (actual size * stripe count) over all. If its working fine for other type of volumes, you can close it. We can work on striped volume in a separate bug. Regards, Amar
For a dist-rep volume [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 limit-usage / 1MB limit set on / [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1024000 [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1024001 [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1025001 [root@centos-qa-client-2 sbin]# ###################### client side##########################3 over nfs mount [root@centos-qa-client-3 nfs-test]# [root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.1 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.141628 seconds, 3.6 MB/s [root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.2 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.231093 seconds, 2.2 MB/s [root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.3 bs=1KB count=200 dd: closing output file `f.3': Disk quota exceeded [root@centos-qa-client-3 nfs-test]# ls -lia total 1060 1 drwxr-xr-x 2 root root 16384 Apr 17 02:18 . 4620289 drwxr-xr-x 23 root root 4096 Apr 17 02:16 .. 10423406667973675939 -rw-r--r-- 1 root root 512000 Apr 17 02:18 f.1 671383396545206460 -rw-r--r-- 1 root root 512000 Apr 17 02:18 f.2 4154998297473212558 -rw-r--r-- 1 root root 0 Apr 17 02:18 f.3 [root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.4 bs=1 count=1 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.02796 seconds, 0.0 kB/s [root@centos-qa-client-3 nfs-test]# ls f.1 f.2 f.3 f.4 [root@centos-qa-client-3 nfs-test]# ls -li total 1028 10423406667973675939 -rw-r--r-- 1 root root 512000 Apr 17 02:18 f.1 671383396545206460 -rw-r--r-- 1 root root 512000 Apr 17 02:18 f.2 4154998297473212558 -rw-r--r-- 1 root root 0 Apr 17 02:18 f.3 12536810315066991238 -rw-r--r-- 1 root root 1 Apr 17 02:19 f.4 [root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.5 bs=1KB count=1 1+0 records in 1+0 records out 1000 bytes (1.0 kB) copied, 0.02385 seconds, 41.9 kB/s [root@centos-qa-client-3 nfs-test]# ls -l total 1036 -rw-r--r-- 1 root root 512000 Apr 17 02:18 f.1 -rw-r--r-- 1 root root 512000 Apr 17 02:18 f.2 -rw-r--r-- 1 root root 0 Apr 17 02:18 f.3 -rw-r--r-- 1 root root 1 Apr 17 02:19 f.4 -rw-r--r-- 1 root root 1000 Apr 17 02:20 f.5 [root@centos-qa-client-3 nfs-test]# So over nfs it is working, will update the result for fuse also
test done over fuse mount for a dist-rep volume [root@centos-qa-client-3 gluster-test]# rm -rf * [root@centos-qa-client-3 gluster-test]# dd if=/dev/zero of=f.1 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.210623 seconds, 2.4 MB/s [root@centos-qa-client-3 gluster-test]# dd if=/dev/zero of=f.2 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.170953 seconds, 3.0 MB/s [root@centos-qa-client-3 gluster-test]# dd if=/dev/zero of=f.3 bs=1KB count=200 dd: closing output file `f.3': Disk quota exceeded [root@centos-qa-client-3 gluster-test]# dd if=/dev/zero of=f.4 bs=1KB count=1 1+0 records in 1+0 records out 1000 bytes (1.0 kB) copied, 0.000647 seconds, 1.5 MB/s [root@centos-qa-client-3 gluster-test]# ls -l total 1044 -rw-r--r-- 1 root root 512000 Apr 17 02:48 f.1 -rw-r--r-- 1 root root 512000 Apr 17 02:48 f.2 -rw-r--r-- 1 root root 14000 Apr 17 02:48 f.3 -rw-r--r-- 1 root root 1000 Apr 17 02:48 f.4 [root@centos-qa-client-3 gluster-test]# ######################################################################## [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 0 [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1024000 [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 1039000 [root@centos-qa-client-2 sbin]# Hence moving this bug to verified state
though I moved this bug to verified state, but I am re-opening as in one of the scenarios of dist-rep volume it still failed. 1. set some limit on volume. 2. create dirs inside it. 3. put some limits on them.one dir having limit larger than that of the volume. 4. create some files inside the dirs. 5. Though the volume limit is not getting crossed, still even if there is space for as per volume and dir limit the files is not getting created. Hence moving bug back to re-opened state, logs, [root@centos-qa-client-3 gluster-test]# rm -rf * [root@centos-qa-client-3 gluster-test]# ls [root@centos-qa-client-3 gluster-test]# mkdir d1 d2 [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 0 /d1 512000 0 /d2 2097152 0 [root@centos-qa-client-3 gluster-test]# cd d2 [root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.1 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.016852 seconds, 30.4 MB/s [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 512000 /d1 512000 0 /d2 2097152 512000 [root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.1 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.021466 seconds, 23.9 MB/s [root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.2 bs=1KB count=512 512+0 records in 512+0 records out 512000 bytes (512 kB) copied, 0.212761 seconds, 2.4 MB/s [root@centos-qa-client-3 d2]# [root@centos-qa-client-3 d2]# [root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.2 bs=1KB count=512 dd: writing `f.2': Disk quota exceeded 4+0 records in 3+0 records out 3000 bytes (3.0 kB) copied, 0.001455 seconds, 2.1 MB/s [root@centos-qa-client-3 d2]# ls -l total 512 -rw-r--r-- 1 root root 512000 Apr 17 03:16 f.1 -rw-r--r-- 1 root root 0 Apr 17 03:16 f.2 [root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.3 bs=1KB count=512 dd: opening `f.3': Disk quota exceeded [root@centos-qa-client-3 d2]# ls -l total 512 -rw-r--r-- 1 root root 512000 Apr 17 03:16 f.1 -rw-r--r-- 1 root root 0 Apr 17 03:16 f.2 [root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.3 bs=1KB count=200 dd: opening `f.3': Disk quota exceeded [root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.4 bs=1KB count=200 dd: opening `f.4': Disk quota exceeded [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 512000 /d1 512000 0 /d2 2097152 512000 [root@centos-qa-client-3 d2]# cd ../d1 [root@centos-qa-client-3 d1]# dd if=/dev/zero of=f.1 bs=1KB count=200 200+0 records in 200+0 records out 200000 bytes (200 kB) copied, 0.008668 seconds, 23.1 MB/s [root@centos-qa-client-3 d1]# [root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list path limit_set size ---------------------------------------------------------------------------------- / 1048576 712000 /d1 512000 200000 /d2 2097152 512000
The issue is not reproducible on my local setup.
patch sent, waiting for it to be accepted.
PATCH: http://patches.gluster.com/patch/7430 in master (features/quota: after validation on a path, reset variable just_validated to zero before checking the size of parent.)