Bug 800649 - volume quota not working
volume quota not working
Status: CLOSED DUPLICATE of bug 765478
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
1.0
x86_64 Linux
high Severity high
: Beta
: ---
Assigned To: Junaid
Saurabh
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-06 16:00 EST by Amaya Rosa Gil Pippino
Modified: 2016-01-19 01:09 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-13 04:28:54 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Amaya Rosa Gil Pippino 2012-03-06 16:00:32 EST
Description of problem:
As per the guide, you can set quotas at directory level or at volume level. I did at volume level, by issuing:
[root@gluster1 ~]# gluster volume quota vol_gl_shared enable
[root@gluster1 ~]# gluster volume quota vol_gl_shared limit-usage / 1GB

As per the guide, you can do # gluster volume quota VOLNAME limit-usage /directorylimit-value
to be also able to set a quota on a given directory.

Nothing else is said on the guide, thus as per my understanding, if I was to write on the mount point on my client machine, quotas may be preventing me to exceed the 1Gb limit. However this does not work.

As per my understanding, quotas only affect a gluster mount point, that is the client, right? And when I try to write to the client, quotas are not hit:
[root@gluster3 vol_geo]# grep gluster /etc/fstab 
gluster1:/vol_gl_shared    /gl_vol               glusterfs       defaults,acl,_netdev 0 0

[root@gluster3 ~]# df -kh
S.ficheros            Size  Used Avail Use% Montado en
/dev/mapper/VolGroup-lv_root
                      1,7G  1,3G  352M  79% /
tmpfs                 246M     0  246M   0% /dev/shm
/dev/vda1             485M   50M  411M  11% /boot
gluster1:/vol_gl_shared
                      3,4G  2,6G  628M  81% /gl_vol	>>>>>>> quoted FS
gluster1:/vol_gl_geo  1,7G  1,4G  279M  83% /mnt/vol_geo

[root@gluster1 ~]# gluster volume quota vol_gl_shared limit-usage / 500MB
limit set on /
[root@gluster1 ~]# gluster volume quota vol_gl_shared list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                         500MB                4.0KB

[root@gluster1 ~]# gluster volume info vol_gl_shared
Volume Name: vol_gl_shared
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/gl_brick
Brick2: gluster2:/gl_brick
Options Reconfigured:
auth.allow: 10.*
cluster.min-free-disk: 15
server.allow-insecure: off
features.quota: on
features.limit-usage: /:200MB
features.quota-timeout: 5

[root@gluster3 ~]# dd if=/dev/urandom of=/gl_vol/kk bs=8192 count=262144
Every 2,0s: ls -lah /gl_brick/                                                                                     Tue Mar  6 20:24:06 2012
-rw-r--r--.  1 root root 436M mar  6 20:22 kk


Version-Release number of selected component (if applicable):
RHSSA 3.2

How reproducible:
Just to configure volumes as previously indicated

Steps to Reproduce:
1. Add a quota to a volume
2. Write a bigger file to the mount point than the quota limit
3.
  
Actual results:
Quota exceeded, file actually created

Expected results:
Quota to prevent file to exceed max size

Additional info:
Comment 1 Amaya Rosa Gil Pippino 2012-03-06 16:18:50 EST
There is a typo on the description, quota is set to 200Mb and not 500Mb:

[root@gluster2 ~]# gluster volume quota vol_gl_shared list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                         200MB                4.0KB
Comment 2 Amaya Rosa Gil Pippino 2012-03-06 16:39:39 EST
Also tried with a regular user, to discard it is affecting only root:

[root@gluster3 vol_geo]# setfacl -m u:amaya:rwx /gl_vol
[root@gluster3 vol_geo]# getfacl /gl_vol
getfacl: Eliminando '/' inicial en nombres de ruta absolutos
# file: gl_vol
# owner: root
# group: root
user::rwx
user:amaya:rwx
group::r-x
mask::rwx
other::rw-

[root@gluster3 vol_geo]# su - amaya

[amaya@gluster3 ~]$ dd if=/dev/urandom of=/gl_vol/kk bs=8192 count=262144
^C35963+0 records in
35963+0 records out
294608896 bytes (295 MB) copied, 53.887 s, 5.5 MB/s

I've stopped dd once quota has been exceeded:
[root@gluster1 ~]# gluster volume info vol_gl_shared 

Volume Name: vol_gl_shared
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/gl_brick
Brick2: gluster2:/gl_brick
Options Reconfigured:
features.limit-usage: /:200MB
features.quota-timeout: 5
features.quota: on
server.allow-insecure: off
cluster.min-free-disk: 15
auth.allow: 10.*
Comment 3 Amar Tumballi 2012-03-19 11:17:59 EDT
There were multiple fixes related to quota through Community->GlusterFS related Bugs, mostly this should be fixed by qa29 release tarball.
Comment 4 Amaya Rosa Gil Pippino 2012-04-09 07:42:05 EDT
Not sure if this is the right place or if I should open a new bug, or if now this has been changed, but now, when trying to create a volume with bricks on different servers but with the same directory name (just like the ones used in this bug data) an error appears, not a possibility any longer, now directory names must be different.

[root@gluster-srv1 ~]# gluster volume create vol_gl_shared gluster-srv1:/gl_brick
Creation of volume vol_gl_shared has been successful. Please start the volume to access data.
[root@gluster-srv1 ~]# gluster volume start vol_gl_shared
Starting volume vol_gl_shared has been successful
[root@gluster-srv1 ~]# gluster peer probe gluster-srv2
Probe successful
[root@gluster-srv1 ~]# gluster volume add-brick vol_gl_shared gluster-srv2:/gl_brick
Brick: gluster-srv2:/gl_brick already in use

Is this a bug or the new way to configure RHSSA?
Comment 6 Sachin Ghai 2012-05-31 04:50:58 EDT
Verified this issue with following latest rpm's from brew:

glusterfs-fuse-3.3.0qa45-1.el6.x86_64
glusterfs-3.3.0qa45-1.el6.x86_64
glusterfs-server-3.3.0qa45-1.el6.x86_64
glusterfs-rdma-3.3.0qa45-1.el6.x86_64
org.apache.hadoop.fs.glusterfs-glusterfs-0.20.2_0.1-1.noarch
glusterfs-geo-replication-3.3.0qa45-1.el6.x86_64

And the reported issue is reproducible. I can write on specified volume even after exceeding the quota limit:


Enabled quota on sghai_vol:
=================
[root@dhcp201-154 glusterfs]# gluster volume quota sghai_vol enable
Enabling quota has been successful

set the quota limit 512 MB:
============================
[root@dhcp201-154 glusterfs]# gluster volume quota sghai_vol limit-usage /home 512MB
limit set on /home


vol info:
==========
[root@dhcp201-154 brick2]# gluster volume info sghai_vol
 
Volume Name: sghai_vol
Type: Distribute
Volume ID: d87e0965-cb29-4ccb-891f-59e65d768944
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: dhcp201-154.englab.pnq.redhat.com:/home/brick1
Brick2: dhcp201-154.englab.pnq.redhat.com:/home/brick2
Options Reconfigured:
features.limit-usage: /home:512MB
features.quota: on

[root@dhcp201-154 glusterfs]# gluster volume quota sghai_vol list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/home                     512MB


on client:
===========

mounted volume on client:
========================

[root@dhcp201-208 ~]# mount -t glusterfs 10.65.201.154:/sghai_vol /mnt

created a file exceeding the quota limit of 512MB

[root@dhcp201-208 mnt]# dd if=/dev/urandom of=/mnt/kk bs=8192 count=262144
^C74091+1 records in
74091+0 records out
606953472 bytes (607 MB) copied, 213.222 s, 2.8 MB/s

[root@dhcp201-208 mnt]# ll -h
total 536M
-rw-r--r--. 1 root root 536M May 31  2012 kk
[root@dhcp201-208 mnt]# 

No error appears neither on server not on client



mounted glusterfs on client:
=============================

[root@dhcp201-208 mnt]# mount
/dev/mapper/vg_dhcp201208-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/vda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
10.65.201.154:/sghai_vol on /mnt type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) >>>>glusterfs, mounted sghai_vol on /mnt
[root@dhcp201-208 mnt]#
Comment 7 Saurabh 2012-05-31 05:37:56 EDT
While trying to verify this bug with 3.3.0qa45, in order to find that the limits do not get crossed, 
 
hence mounted the xfs with the option allocsize=4096 on all the servers.

put up a quota limit of 1GB over volume(/)
now tried to create a file of size 1GB from the glusterfs mount.

the issue happens is that file that gets created is only of around 500MB(approx)
whereas the disk quota exceeded limit is notified.

At this time the "quota list" displays the size field as "1GB". Though after sometime it displays the correct size.

To me this is a issue, if the disk quota disk limit gets hit half way of reaching the limit actually set.

logs,

[root@QA-31 glfs]# dd if=/dev/zero of=f.1 bs=1024 count=1048576
dd: writing `f.1': Disk quota exceeded
dd: closing output file `f.1': Disk quota exceeded
[root@QA-31 glfs]# ls -l
total 524456
-rw-r--r--. 1 root root 537038848 May 31 05:37 f.1
[root@QA-31 glfs]# 


[root@RHS-71 ~]# gluster volume quota dist-rep list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           1GB                1.0GB

after a delay,

[root@RHS-71 ~]# gluster volume quota dist-rep list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           1GB              512.2MB
Comment 8 Amar Tumballi 2012-05-31 05:53:11 EDT
Sachin, about comment #6, when you set quota on '/home',  you need to create files on '/mnt/home' if glusterfs is mounted on /mnt.

Can you verify bug having that in mind?
Comment 9 Sudhir D 2012-06-08 11:03:00 EDT
Moving to ON_QA to be verify with Amar's comment #8
Comment 10 Saurabh 2012-06-11 08:44:10 EDT
creating a 2GB file from fuse mount :-

===================================
while quota limit set is 1GB on root of the volume,

file gets created using dd.

the size of the file is less around 512MB but still quota returns with quota limit reached message

whereas after it starts showing correct value for the size occupied.

But full file not getting creating is an issue over fuse mount

[root@QA-31 glfs]# rm -rf *
[root@QA-31 glfs]# dd if=/dev/zero of=f.1 bs=1048576 count=2048
dd: writing `f.1': Disk quota exceeded
dd: closing output file `f.1': Disk quota exceeded
[root@QA-31 glfs]# du -h .
513M	.


[root@RHS-71 ~]# gluster volume quota dist-rep1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           1GB               0Bytes
[root@RHS-71 ~]# gluster volume quota dist-rep1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           1GB                1.0GB
[root@RHS-71 ~]# gluster volume quota dist-rep1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           1GB              512.6MB




creating file from nfs mount:-
============================
on nfs the same issue is not seen, as mentioned above for fuse

1. file of size 2GB while 1GB is the quota limit-set over volume.

[root@QA-31 nfs-test2]# rm -rf *
[root@QA-31 nfs-test2]# dd if=/dev/zero of=f.1 bs=1048576 count=2048
dd: writing `f.1': Input/output error
706+0 records in
705+0 records out
739246080 bytes (739 MB) copied, 9.69904 s, 76.2 MB/s
[root@QA-31 nfs-test2]# du -h .
1.2G	.
[root@QA-31 nfs-test2]# du -h .
1.2G	.
[root@QA-31 nfs-test2]# 


[root@RHS-71 ~]# 
[root@RHS-71 ~]# gluster volume quota dist-rep1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           1GB                1.2GB
[root@RHS-71 


same is the case for fuse mount even if the xfs mount is done with allocsize=4096
Comment 11 Junaid 2012-06-12 08:10:52 EDT
I ran the dd command as mentioned by Saurabh. In case of ext4 as the backend, this problem is not seen. But in xfs, I see that at some point in time marker translator on the server side got 2098184 blocks being allocated to the file even though the file size is 537522176bytes as part of stat. This can be seen below(it can be seen in ia_blocks and ia_size respectively).

(gdb) p *buf
$3 = {ia_ino = 12404037568965143247, ia_gfid = "\342%\227t5%IÓ¬#\376s\026\304j", <incomplete sequence \317>, ia_dev = 64513, 
  ia_type = IA_IFREG, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 1 '\001', write = 1 '\001', 
      exec = 0 '\000'}, group = {read = 1 '\001', write = 0 '\000', exec = 0 '\000'}, other = {read = 1 '\001', write = 0 '\000', 
      exec = 0 '\000'}}, ia_nlink = 1, ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 537522176, ia_blksize = 4096, 
  ia_blocks = 2098184, ia_atime = 1339502522, ia_atime_nsec = 26682256, ia_mtime = 1339502525, ia_mtime_nsec = 876682676, 
  ia_ctime = 1339502525, ia_ctime_nsec = 876682676}

So, what I can conclude for now is that XFS is pre-allocating the blocks and quota uses blocks*512 to calculate the size of the file. Hence, the error.
Comment 12 Junaid 2012-06-13 04:28:54 EDT

*** This bug has been marked as a duplicate of bug 765478 ***

Note You need to log in before you can comment on or make changes to this bug.