Bug 764473 (GLUSTER-2741)

Summary:	Quota: new files not getting created even if space is available
Product:	[Community] GlusterFS	Reporter:	Saurabh <saurabh>
Component:	quota	Assignee:	Raghavendra G <raghavendra>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	mainline	CC:	amarts, gluster-bugs
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Saurabh 2011-04-13 06:41:03 UTC

Same issue happened in this scenario also,


[root@centos-qa-client-1 gluster-test]# ls -R
.:
d1  d2

./d1:
dd1

./d1/dd1:

./d2:
dd2

./d2/dd2:
[root@centos-qa-client-1 gluster-test]# cd d1/dd1
[root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=512
dd: closing output file `f.1': Disk quota exceeded
[root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=500
dd: writing `f.1': Disk quota exceeded
dd: closing output file `f.1': Disk quota exceeded
[root@centos-qa-client-1 dd1]# ls -li
total 4
18481504 -rw-r--r-- 1 root root 0 Apr 13 02:31 f.1
[root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.1 bs=1KB count=200
dd: writing `f.1': Disk quota exceeded
dd: closing output file `f.1': Disk quota exceeded
[root@centos-qa-client-1 dd1]# rm -rf *
[root@centos-qa-client-1 dd1]# dd if=/dev/zero of=f.2 bs=1KB count=400
dd: opening `f.2': Disk quota exceeded
[root@centos-qa-client-1 dd1]# ls -lia
total 64
18481468 drwxr-xr-x 2 root root 16384 Apr 13 02:31 .
18481440 drwxr-xr-x 3 root root 16384 Apr 13 02:29 ..
[root@centos-qa-client-1 dd1]# 

####################################
[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-rep2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576                    0
/d1/dd1                  512000                    0
/d2                     1048576                    0
/d2/dd2                  512000                    0
[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-rep2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576                    0
/d1/dd1                  512000                    0
/d2                     1048576                    0
/d2/dd2                  512000                    0
[root@centos-qa-client-2 sbin]#

Comment 1 Saurabh 2011-04-13 08:24:10 UTC

Setup,

  Vol: Dist-strip,
  Quota: on 
  Quota limit set: on a diretory and the root not having any limit set.

  test,

  1. create a dir over fuse mount.
  2. set the quota limit to 1MB.
  3. create a file using "dd if=/dev/zero of=f.1 bs=1KB count=512"
  4. create one more file using same command.
  5. Now use the same dd command as in step4 for creating the same file again.
  6. try creating some more files, 
  

  Expected Result:- the steps 5,6 should be success
  Actual Result:- The steps 5,6 are failure.


  Logs,

  From fuse client,

 [root@centos-qa-client-1 g-diststrip]# cd d1
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.2 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.209521 seconds, 2.4 MB/s
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.2 bs=1KB count=100
dd: writing `f.2': Disk quota exceeded
dd: closing output file `f.2': Disk quota exceeded
[root@centos-qa-client-1 d1]# ls -li
total 512
9240706 -rw-r--r-- 1 root root 512000 Apr 13 00:03 f.1
9240604 -rw-r--r-- 1 root root      0 Apr 13 01:06 f.2
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.2 bs=1KB count=512
dd: writing `f.2': Disk quota exceeded
dd: closing output file `f.2': Disk quota exceeded
[root@centos-qa-client-1 d1]# rm f.2
rm: remove regular empty file `f.2'? n
[root@centos-qa-client-1 d1]# ls -li
total 512
9240706 -rw-r--r-- 1 root root 512000 Apr 13 00:03 f.1
9240604 -rw-r--r-- 1 root root      0 Apr 13 01:07 f.2
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1KB count=512
dd: opening `f.3': Disk quota exceeded
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1KB count=1
dd: opening `f.3': Disk quota exceeded
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1 count=1
dd: opening `f.3': Disk quota exceeded
[root@centos-qa-client-1 d1]# ls -li
total 512
9240706 -rw-r--r-- 1 root root 512000 Apr 13 00:03 f.1
9240604 -rw-r--r-- 1 root root      0 Apr 13 01:07 f.2
[root@centos-qa-client-1 d1]# rm f.2
rm: remove regular empty file `f.2'? y
[root@centos-qa-client-1 d1]# dd if=/dev/zero of=f.3 bs=1 count=1
dd: opening `f.3': Disk quota exceeded
[root@centos-qa-client-1 d1]# ls -li
total 508
9240706 -rw-r--r-- 1 root root 512000 Apr 13 00:03 f.1
[root@centos-qa-client-1 d1]# pwd
/mnt/g-diststrip/d1
[root@centos-qa-client-1 d1]# mount | strip1
-bash: strip1: command not found
[root@centos-qa-client-1 d1]# mount | grep strip1
glusterfs#10.1.12.134:/dist-strip1 on /mnt/g-diststrip type fuse (rw,allow_other,default_permissions,max_read=131072)
[root@centos-qa-client-1 d1]# 
[root@centos-qa-client-1 d1]# 


#########################################################3


from server end,

[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576               512000
[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576              1024000
[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576               512000
[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576               512000
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dist-strip1/d1/f.1
getfattr: Removing leading '/' from absolute path names
# file: mnt/dist-strip1/d1/f.1
trusted.gfid=0xc44457bfab5744fc8e04b5c6ba0459e2
trusted.glusterfs.quota.11126cff-4e45-47d3-93ba-0cefb5b97a11.contri=0x000000000007d000

[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dist-strip1/d1/f.2
getfattr: Removing leading '/' from absolute path names
# file: mnt/dist-strip1/d1/f.2
trusted.gfid=0xdefa4d77d89047238d85558fdedc9d92
trusted.glusterfs.quota.11126cff-4e45-47d3-93ba-0cefb5b97a11.contri=0x0000000000000000

[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576               512000
[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576               512000
[root@centos-qa-client-2 sbin]# getfattr -m . -d -e hex /mnt/dist-strip1/
getfattr: Removing leading '/' from absolute path names
# file: mnt/dist-strip1
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x000000000007d000
trusted.glusterfs.test=0x776f726b696e6700

[root@centos-qa-client-2 sbin]# ./gluster volume quota dist-strip1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576               512000
[root@centos-qa-client-2 sbin]#

Comment 2 Anand Avati 2011-04-14 05:00:53 UTC

PATCH: http://patches.gluster.com/patch/6864 in master (quota/(f)truncate: pass correct delta for updating parent sizes after successful completion.)

Comment 3 Saurabh 2011-04-14 11:07:39 UTC

I am having qa-12 release for glusterfs 

and this issue again happening in this kind of envrionment,

Well, the volume is a striped volume.



[root@centos-qa-client-1 g-ds2]# mkdir d3
[root@centos-qa-client-1 g-ds2]# mkdir d3/dd3
[root@centos-qa-client-1 g-ds2]# cd d3/dd3
[root@centos-qa-client-1 dd3]# ls
[root@centos-qa-client-1 dd3]# dd if=/dev/zero of=f.1 bs=1KB count=500
500+0 records in
500+0 records out
500000 bytes (500 kB) copied, 0.170084 seconds, 2.9 MB/s
[root@centos-qa-client-1 dd3]# cd ..
[root@centos-qa-client-1 d3]# rm -rf dd3
[root@centos-qa-client-1 d3]# dd if=/dev/zero of=f.1 bs=1KB count=500
dd: closing output file `f.1': Disk quota exceeded
[root@centos-qa-client-1 d3]# ls -li
total 136
4621122 -rw-r--r-- 1 root root 458000 Apr 14 06:50 f.1
[root@centos-qa-client-1 d3]# cd ..
[root@centos-qa-client-1 g-ds2]# mkdir d4
[root@centos-qa-client-1 g-ds2]# mkdir d4/dd4
[root@centos-qa-client-1 g-ds2]# cd d4/dd4
[root@centos-qa-client-1 dd4]# dd if=/dev/zero of=f.1 bs=1KB count=500
500+0 records in
500+0 records out
500000 bytes (500 kB) copied, 0.168528 seconds, 3.0 MB/s
[root@centos-qa-client-1 dd4]# cd ..
[root@centos-qa-client-1 d4]# rm -rf dd4
[root@centos-qa-client-1 d4]# dd if=/dev/zero of=f.1 bs=1KB count=500
500+0 records in
500+0 records out
500000 bytes (500 kB) copied, 0.168964 seconds, 3.0 MB/s
[root@centos-qa-client-1 d4]# ls -li
total 136
4621128 -rw-r--r-- 1 root root 500000 Apr 14 07:01 f.1
[root@centos-qa-client-1 d4]# dd if=/dev/zero of=f.2 bs=1KB count=500
dd: opening `f.2': Disk quota exceeded
[root@centos-qa-client-1 d4]# ls -li
total 136
4621128 -rw-r--r-- 1 root root 500000 Apr 14 07:01 f.1
[root@centos-qa-client-1 d4]# dd if=/dev/zero of=f.2 bs=1KB count=300
dd: opening `f.2': Disk quota exceeded
[root@centos-qa-client-1 d4]# ls -li
total 136
4621128 -rw-r--r-- 1 root root 500000 Apr 14 07:01 f.1
[root@centos-qa-client-1 d4]# dd if=/dev/zero of=f.2 bs=1KB count=1
dd: opening `f.2': Disk quota exceeded
[root@centos-qa-client-1 d4]# ls -li
total 136
4621128 -rw-r--r-- 1 root root 500000 Apr 14 07:01 f.1
[root@centos-qa-client-1 d4]# 



#######################################
[root@centos-qa-client-2 sbin]# ./gluster volume quota ds2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/d1                     1048576               131072
/d1/dd1                  512000               131072
/d2/dd2                   500KB
/d3                     1048576               131072
/d3/dd3                   500KB
/d4                     1048576               131072
/d4/dd4                   500KB


The size are not shown correctly because of bug764492,

Comment 4 Amar Tumballi 2011-04-14 12:54:09 UTC

Saurabh,

Striped volumes have some known issues with quota (as marker works on st_size instead of st_blocks).

Striped volumes will consume (actual size * stripe count) over all.

If its working fine for other type of volumes, you can close it. We can work on striped volume in a separate bug.

Regards,
Amar

Comment 5 Saurabh 2011-04-17 06:23:09 UTC

For a dist-rep volume

[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 limit-usage / 1MB
limit set on /
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576              1024000
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576              1024001
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576              1025001
[root@centos-qa-client-2 sbin]# 



###################### client side##########################3
over nfs mount

[root@centos-qa-client-3 nfs-test]# 
[root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.1 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.141628 seconds, 3.6 MB/s
[root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.2 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.231093 seconds, 2.2 MB/s
[root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.3 bs=1KB count=200
dd: closing output file `f.3': Disk quota exceeded
[root@centos-qa-client-3 nfs-test]# ls -lia
total 1060
                   1 drwxr-xr-x  2 root root  16384 Apr 17 02:18 .
             4620289 drwxr-xr-x 23 root root   4096 Apr 17 02:16 ..
10423406667973675939 -rw-r--r--  1 root root 512000 Apr 17 02:18 f.1
  671383396545206460 -rw-r--r--  1 root root 512000 Apr 17 02:18 f.2
 4154998297473212558 -rw-r--r--  1 root root      0 Apr 17 02:18 f.3
[root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.4 bs=1 count=1
1+0 records in
1+0 records out
1 byte (1 B) copied, 0.02796 seconds, 0.0 kB/s
[root@centos-qa-client-3 nfs-test]# ls
f.1  f.2  f.3  f.4
[root@centos-qa-client-3 nfs-test]# ls -li
total 1028
10423406667973675939 -rw-r--r-- 1 root root 512000 Apr 17 02:18 f.1
  671383396545206460 -rw-r--r-- 1 root root 512000 Apr 17 02:18 f.2
 4154998297473212558 -rw-r--r-- 1 root root      0 Apr 17 02:18 f.3
12536810315066991238 -rw-r--r-- 1 root root      1 Apr 17 02:19 f.4
[root@centos-qa-client-3 nfs-test]# dd if=/dev/zero of=f.5 bs=1KB count=1
1+0 records in
1+0 records out
1000 bytes (1.0 kB) copied, 0.02385 seconds, 41.9 kB/s
[root@centos-qa-client-3 nfs-test]# ls -l
total 1036
-rw-r--r-- 1 root root 512000 Apr 17 02:18 f.1
-rw-r--r-- 1 root root 512000 Apr 17 02:18 f.2
-rw-r--r-- 1 root root      0 Apr 17 02:18 f.3
-rw-r--r-- 1 root root      1 Apr 17 02:19 f.4
-rw-r--r-- 1 root root   1000 Apr 17 02:20 f.5
[root@centos-qa-client-3 nfs-test]# 


So over nfs it is working, will update the result for fuse also

Comment 6 Saurabh 2011-04-17 06:55:06 UTC

test done over fuse mount for a dist-rep volume

[root@centos-qa-client-3 gluster-test]# rm -rf *
[root@centos-qa-client-3 gluster-test]# dd if=/dev/zero of=f.1 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.210623 seconds, 2.4 MB/s
[root@centos-qa-client-3 gluster-test]# dd if=/dev/zero of=f.2 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.170953 seconds, 3.0 MB/s
[root@centos-qa-client-3 gluster-test]# dd if=/dev/zero of=f.3 bs=1KB count=200
dd: closing output file `f.3': Disk quota exceeded
[root@centos-qa-client-3 gluster-test]# dd if=/dev/zero of=f.4 bs=1KB count=1
1+0 records in
1+0 records out
1000 bytes (1.0 kB) copied, 0.000647 seconds, 1.5 MB/s
[root@centos-qa-client-3 gluster-test]# ls -l
total 1044
-rw-r--r-- 1 root root 512000 Apr 17 02:48 f.1
-rw-r--r-- 1 root root 512000 Apr 17 02:48 f.2
-rw-r--r-- 1 root root  14000 Apr 17 02:48 f.3
-rw-r--r-- 1 root root   1000 Apr 17 02:48 f.4
[root@centos-qa-client-3 gluster-test]# 


########################################################################


[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
    path          limit_set         size
----------------------------------------------------------------------------------
/                       1048576                    0
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
    path          limit_set         size
----------------------------------------------------------------------------------
/                       1048576              1024000
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
    path          limit_set         size
----------------------------------------------------------------------------------
/                       1048576              1039000
[root@centos-qa-client-2 sbin]# 


Hence moving this bug to verified state

Comment 7 Saurabh 2011-04-17 07:30:56 UTC

though I moved this bug to verified state, but I am re-opening as in one of the scenarios of dist-rep volume it still failed.

1. set some limit on volume.
2. create dirs inside it.
3. put some limits on them.one dir having limit larger than that of the volume.
4. create some files inside the dirs.
5. Though the volume limit is not getting crossed, still even if there is space for as per volume and dir limit the files is not getting created.


  Hence moving bug back to re-opened state,

logs,


[root@centos-qa-client-3 gluster-test]# rm -rf *
[root@centos-qa-client-3 gluster-test]# ls
[root@centos-qa-client-3 gluster-test]# mkdir d1 d2

[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576                    0
/d1                      512000                    0
/d2                     2097152                    0


[root@centos-qa-client-3 gluster-test]# cd d2
[root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.1 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.016852 seconds, 30.4 MB/s
[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576               512000
/d1                      512000                    0
/d2                     2097152               512000



[root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.1 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.021466 seconds, 23.9 MB/s
[root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.2 bs=1KB count=512
512+0 records in
512+0 records out
512000 bytes (512 kB) copied, 0.212761 seconds, 2.4 MB/s
[root@centos-qa-client-3 d2]# 
[root@centos-qa-client-3 d2]# 
[root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.2 bs=1KB count=512
dd: writing `f.2': Disk quota exceeded
4+0 records in
3+0 records out
3000 bytes (3.0 kB) copied, 0.001455 seconds, 2.1 MB/s
[root@centos-qa-client-3 d2]# ls -l
total 512
-rw-r--r-- 1 root root 512000 Apr 17 03:16 f.1
-rw-r--r-- 1 root root      0 Apr 17 03:16 f.2
[root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.3 bs=1KB count=512
dd: opening `f.3': Disk quota exceeded
[root@centos-qa-client-3 d2]# ls -l
total 512
-rw-r--r-- 1 root root 512000 Apr 17 03:16 f.1
-rw-r--r-- 1 root root      0 Apr 17 03:16 f.2
[root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.3 bs=1KB count=200
dd: opening `f.3': Disk quota exceeded
[root@centos-qa-client-3 d2]# dd if=/dev/zero of=f.4 bs=1KB count=200
dd: opening `f.4': Disk quota exceeded

[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576               512000
/d1                      512000                    0
/d2                     2097152               512000

[root@centos-qa-client-3 d2]# cd ../d1
[root@centos-qa-client-3 d1]# dd if=/dev/zero of=f.1 bs=1KB count=200
200+0 records in
200+0 records out
200000 bytes (200 kB) copied, 0.008668 seconds, 23.1 MB/s
[root@centos-qa-client-3 d1]# 


[root@centos-qa-client-2 sbin]# ./gluster volume quota dr2 list 
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                       1048576               712000
/d1                      512000               200000
/d2                     2097152               512000

Comment 8 Raghavendra G 2011-04-19 09:05:00 UTC

The issue is not reproducible on my local setup.

Comment 9 Raghavendra G 2011-06-09 02:23:48 UTC

patch sent, waiting for it to be accepted.

Comment 10 Anand Avati 2011-06-16 13:16:02 UTC

PATCH: http://patches.gluster.com/patch/7430 in master (features/quota: after validation on a path, reset variable just_validated to zero before checking the size of parent.)