Hi Ramana, To set the limit on directory "dheeraj" which is under the mount-point "/testgluster" run gluster volume quota vol-name limit-usage /dheeraj 5GB The point is for gluster-quota the directories are relative the mount-point. Hope this helps. Junaid
Hi Junaid, Thank you for the immediate reply. As suggested by you we had set the usage list using the following command gluster volume quota testfs01 limit-usage /dheeraj 5GB It also lists the same on the volume info command output as attached ~]# gluster volume info Volume Name: testfs01 Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: gs02-ib:/data/gluster/brick-2 Brick2: gs01-ib:/data/gluster/brick-1 Options Reconfigured: features.quota: on diagnostics.count-fop-hits: on diagnostics.latency-measurement: on features.limit-usage: /testgluster/dheeraj:5GB We were able to list the gluster volume usage # gluster volume quota testfs01 list path limit_set size -------------------------------------------------------------------------------- /dheeraj 5368709120 6091304960 After which we had copied some data on the spicified directory, but the quota does not restrict copying data. The above output show the same. Regards, Ramana Kasaraneni.
Hi, We are having a strange problem, while using the quotas in GlusterFS. We are using Gluster 3.2. We have configured gluster on Storage nodes. # rpm -qa |grep glust glusterfs-fuse-3.2.0-1 glusterfs-core-3.2.0-1 glusterfs-rdma-3.2.0-1 The glusterFS configuration looks as follows: [root@gs01 ~]# gluster volume info Volume Name: testfs01 Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: gs02-ib:/data/gluster/brick-2 Brick2: gs01-ib:/data/gluster/brick-1 Options Reconfigured: features.quota: on diagnostics.count-fop-hits: on diagnostics.latency-measurement: on features.limit-usage: /testgluster/dheeraj:5GB We have configured the gluster quota, at the directory level. It does not list the usage on quota enabled directory. # gluster volume quota testfs01 list path limit_set size --------------------------------------------------------------------------- /testgluster/dheeraj 5GB We tried copying more data on to the specified folder, more than 5 GB, and we are able to write more data on the file system. Request you to provide some inputs on the same. On the client nodes we had mounted the gluster file system [root@n0 testgluster]# df -h Filesystem Size Used Avail Use% Mounted on /dev/cciss/c0d0p5 39G 12G 25G 32% / tmpfs 7.9G 380K 7.9G 1% /dev/shm /dev/cciss/c0d0p1 281M 44M 223M 17% /boot /dev/cciss/c0d0p2 21G 14G 6.6G 67% /var 10.1.32.33@o2ib0:/lfs1 135G 112G 17G 88% /lustre 10.1.60.1@o2ib0:/lfs01 68G 22G 43G 35% /testlustre /etc/glusterfs/testfs01.vol 135G 28G 101G 22% /testgluster We want to implement quota on the gluster file system. So request your help on the same. Regards, Ramana Kasaraneni.
Hi Ramana, The problem might be because of the way the client is mounted /etc/glusterfs/testfs01.vol 135G 28G 101G 22% /testgluster Please use mount -t glusterfs server-name:/vol-name mount-point Because if you mount directly from the volfile-file, any further changes to the volume options done through gluster cli after the client mount will not be accessible to the client. So I suspect that the changes are not visible to the client. Also, the output of gluster volume info is inconsisten for the option "features.limit-usage". It should have been features.limit-usage: /dheeraj:5GB > features.quota: on > diagnostics.count-fop-hits: on > diagnostics.latency-measurement: on > features.limit-usage: /testgluster/dheeraj:5GB Can you please check the volume info output again and report it here. Junaid
Hi Junaid, As suggested, we had unmount the file system, and mounted the file system using the following command ]# mount -t glusterfs gs01-ib:/testfs01 /testgluster/ and mounted file system looks like ]# df -h Filesystem Size Used Avail Use% Mounted on /dev/cciss/c0d0p5 39G 12G 25G 32% / tmpfs 7.9G 380K 7.9G 1% /dev/shm /dev/cciss/c0d0p1 281M 44M 223M 17% /boot /dev/cciss/c0d0p2 21G 14G 6.6G 67% /var 10.1.32.33@o2ib0:/lfs1 135G 112G 17G 88% /lustre 10.1.60.1@o2ib0:/lfs01 68G 22G 43G 35% /testlustre gs01-ib:/testfs01 135G 0 135G 0% /testgluster In the above mounting, it show wrong info in the usage part. The actual used space is not getting reflected. # cd /testgluster/ [root@n0 testgluster]# ls a createperl dheeraj iyard01 [root@n0 testgluster]# du -sh * 4.0K a 4.0K createperl 5.8G dheeraj 22G iyard01 The actual usage on this file system is near to 29 GB, but it shows has 0% used. Even after changing the mounting mechanism, the quota does not work. # gluster volume quota testfs01 list path limit_set size -------------------------------------------------------------------------------- /dheeraj 5368709120 6132764672 The volume info also the correct info # gluster volume info Volume Name: testfs01 Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: gs02-ib:/data/gluster/brick-2 Brick2: gs01-ib:/data/gluster/brick-1 Options Reconfigured: diagnostics.latency-measurement: on diagnostics.count-fop-hits: on features.quota: on features.limit-usage: /dheeraj:5GB We suspect the issue seems to be some thing else. Regards, Ramana Kasaraneni.
> [root@n0 testgluster]# du -sh * > 4.0K a > 4.0K createperl > 5.8G dheeraj > 22G iyard01 > > The actual usage on this file system is near to 29 GB, but it shows has 0% > used. > > Even after changing the mounting mechanism, the quota does not work. > > # gluster volume quota testfs01 list > path limit_set size > -------------------------------------------------------------------------------- > /dheeraj 5368709120 6132764672 > Can you please check the behavior using 3.2.3(latest release), some quota bugs have been fixed in it. Both client and server must be updated to the new release.
Need to get information on whether upgrading to new version fixed issues.
quotas still don't work in 3.2.5, I haven't tried 3.2.6. volumne pirdist is a 5 brick distributed cluster and pirstripe is a 5 brick striped cluster. I set the limit for myself to 10GB. First it indicated the size was 0, when there were already 1.2GB of files in the dir. Then I was able to create a single 20GB file in it without the write being stopped at 20GB. gluster> volume quota pirdist list path limit_set size ---------------------------------------------------------------------------------- /user 10GB 20.0GB I thought maybe there was just an issue with me writing a single huge file and expecting it to catch it, so I started writing another 5GB file and it didn't prevent that write either : gluster> volume quota pirdist list path limit_set size ---------------------------------------------------------------------------------- /user 10GB 23.6GB Same issue on pirstripe : gluster> volume quota pirstripe list path limit_set size ---------------------------------------------------------------------------------- /user 5GB 20.0GB ...except here I actually wrote a 10GB file, so it's reporting the size incorrectly, but not in the same way as it was for a "du -sh" where it was reporting the total size as size * 5 (# of storage bricks). The incorrect size report is mentioned in another bug.
ok, looks like the reason the quotas weren't working was because one of the bricks wasn't properly connected to the cluster. However, I have another strange issue where the quota thinks that there's actually 20GB of data in the directory even though there's ~2MB, so it won't let me do any writes. How does the quota determine how much storage a user is actually using? How do I reset this value? I tried removing the quota and adding it again, didn't help. Tried disabling and re-enabling the quotas, etc. gluster> volume quota pirstripe list path limit_set size ---------------------------------------------------------------------------------- /user 2GB 20.0GB du -sh from the fuse mount : % du -sh user 2.0M user ^ I also did a du -sh in the directory at the filesystem layer across all the bricks and it adds up approximately to 2M. again from the fuse mount : % du -sh --apparent-size user 192K user
I have the same issue with Gluster 3.3.1: $ gluster volume quota md0 list Fri Jan 11 16:08:54 2013 path limit_set size ---------------------------------------------------------------------------------- /test 25MB 84.3MB I have these options set on my volume: features.quota-timeout: 0 features.limit-usage: /test:25MB features.quota: on
My problem was caused by mounting GlusterFS with the "-o acl" option, this caused the Quota issue and will be resolved in Gluster 3.4.x https://bugzilla.redhat.com/show_bug.cgi?id=857874 Rich
as per comment #10