Bug 1016478 - quota: dir quota limit mismatches in calculation with actual amount of data in it
quota: dir quota limit mismatches in calculation with actual amount of data i...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.1
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Raghavendra G
Saurabh
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-08 04:55 EDT by Saurabh
Modified: 2016-01-19 01:13 EST (History)
7 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.40rhs-1
Doc Type: Bug Fix
Doc Text:
Previously, the disk usage accounting by quota would go wrong, when rename(3) system call was performed on a volume, with quota enabled and quota limits configured on them. Now, with this update, the disk usage accounting would reflect the disk space used by the volume correctly, even when rename(3) system call was performed by applications running on the volume.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-27 10:41:33 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Test script to check quota accounting in different rename scenarios with distribute (9.81 KB, text/troff)
2013-11-06 07:31 EST, Raghavendra G
no flags Details

  None (edit)
Description Saurabh 2013-10-08 04:55:44 EDT
Description of problem:

tried to mv data one directory(dir1) to another in a directory(dir2) already having some data in it.
dir1 and dir2 both have 2GB quota limit set and reached this limit.


Version-Release number of selected component (if applicable):
glusterfs-3.4.0.34rhs-1.el6rhs.x86_64

How reproducible:
always

Steps to Reproduce:
1. create a volume, start it.

2. enable quota on the volume

3. mount the volume using NFS

4. create two directories inside the mount point.

5. gluster volume quota <vol-name> limit-usage /dir{1-2} 2GB

6. create data inside dir1 till limit is not filled.

7. gluster volume quota <vol-name> list

8. rename all the files inside the dir1.

9. gluster volume quota <vol-name> list

10. copy the data from dir1 to dir2.

11. gluster volume quota <vol-name> list

12. mv data from dir1 to dir2.

13. gluster volume quota <vol-name> list

Actual results:
now here the till step 11 it works, 
the result of step 11 is this,

[root@quota1 ~]# gluster volume quota dist-rep list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir1                                      2.0GB       80%       2.0GB  0Bytes
/dir2                                      2.0GB       80%       2.0GB  64.0KB


now for step 12 and step 13, has a failure,

on client executed this command,
[root@rhsauto030 nfs-test]# mv -f dir1/* dir2/

after this
on rhs-node executed this command,
[root@quota1 ~]# gluster volume quota dist-rep list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir1                                      2.0GB       80%       1.8GB 200.0MB
/dir2                                      2.0GB       80%     648.0MB   1.4GB


whereas in /dir1 there is nothing remaining, as per the ls on mountpoint 
[root@rhsauto030 nfs-test]# ls dir1/
[root@rhsauto030 nfs-test]# 


Though after a long delay, 
quota list respondes this,
[root@quota1 ~]# gluster volume quota dist-rep list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir1                                      2.0GB       80%       1.8GB 200.0MB
/dir2                                      2.0GB       80%       2.0GB  0Bytes


seems there is still a miscalculation for 1.8GB. where as there is nothing inside /dir1.

Expected results:
/dir1 should be zero now.
Also, before all this in this case when both dir1 and dir2 are having quota limits reached, so moving data from dir1 to dir2 should result in "EDQUOT"

Additional info:
Comment 3 Raghavendra G 2013-10-30 23:43:44 EDT
Saurabh,

Can you please provide the exact names of the files before and after rename. Its a distribute volume and names are important since there can be different rename scenarios in dht based on names. It would be better if you copy/paste the exact comments you used.

regards,
Raghavendra.
Comment 4 Saurabh 2013-10-31 00:42:11 EDT
well, most of the relevant information is provided in the description section, and its 20days since this test was executed, and I can't collect the history of the commands from system.

but usually I keep the names like $i, where i represents a numeral.
and rename may be like mv $i $i-rename.
Comment 5 Raghavendra G 2013-11-06 07:31:05 EST
Created attachment 820311 [details]
Test script to check quota accounting in different rename scenarios with distribute
Comment 6 Raghavendra G 2013-11-06 07:33:33 EST
https://code.engineering.redhat.com/gerrit/#/c/15123/
https://code.engineering.redhat.com/gerrit/#/c/14843/
https://code.engineering.redhat.com/gerrit/#/c/14991/

These three fixes collectively should fix accounting issues during different rename scenarios with distribute. The fixes were tested with the script attached in my previous comment.
Comment 7 Saurabh 2013-11-08 06:58:57 EST
I tried out the similar test as mentioned in steps to reproduce section.

filed the directories with data,

[root@quota1 gluster]# gluster volume quota dist-rep list /dir5
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir5                                     50.0GB       80%      50.0GB  0Bytes
[root@quota1 gluster]# gluster volume quota dist-rep list /dir4
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir4                                     50.0GB       80%      50.0GB  0Bytes




mount-point,

[qa1@rhsauto005 nfs-test]$ mount | grep 10.70.42.186:/dist-rep
10.70.42.186:/dist-rep on /mnt/nfs-test type nfs (rw,addr=10.70.42.186)


then tried to mv data from one dir to another one, it says Permission denied, but I assume it should say "Disk quota exceeded"

[qa1@rhsauto005 nfs-test]$ ls -ldi dir4
11242701225515877322 drwxr-xr-x. 2 qa1 qa1 49152 Nov  8 11:20 dir4
[qa1@rhsauto005 nfs-test]$ ls -ldi dir5
12418057083054750413 drwxr-xr-x. 2 qa1 qa1 49152 Nov  8 11:22 dir5
[qa1@rhsauto005 nfs-test]$ 
[qa1@rhsauto005 nfs-test]$ 
[qa1@rhsauto005 nfs-test]$ ls -li dir4/1
11082972577280890575 -rw-rw-r--. 1 qa1 qa1 131072 Nov  8 11:04 dir4/1
[qa1@rhsauto005 nfs-test]$ mv dir4/1 dir5/
mv: cannot move `dir4/1' to `dir5/1': Permission denied



Please, clarify if it suppose to get EPERM, though from my point of view it is incorrect.
Comment 8 Saurabh 2013-11-08 07:49:40 EST
As per discussion with developer  and based on his debugging the Comment7 failure may be happening because of some acl issue.

Presently we are testing the similar stuff on another cluster.
Comment 9 Saurabh 2013-11-08 08:33:56 EST
latest update,

On other set of cluster, I tried to do the similar scenario and didn't see the EPERM rather as expected we got "EDQUOT"

[qa1@rhslong03 nfs-test-dist-rep1]$ mv dir4/1 dir5/
mv: cannot move `dir4/1' to `dir5/1': Disk quota exceeded


after the mv operation, directories in consideration are dir4 and dir5
[root@quota5 ~]# gluster volume quota dist-rep1 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0TB       80%     377.4GB 646.6GB
/qa1                                     200.0GB       80%     224.1GB  0Bytes
/qa2-rename                              300.0GB       75%      0Bytes 300.0GB
/qa3/Downloads                            50.0GB       80%      50.0GB  0Bytes
/dir4                                     50.0GB       80%      50.0GB  0Bytes
/dir5                                     50.0GB       80%      50.0GB  0Bytes
Comment 10 Saurabh 2013-11-08 09:34:25 EST
moving to verified based on comment 9
Comment 11 errata-xmlrpc 2013-11-27 10:41:33 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html
Comment 12 Peter Auyeung 2014-08-29 02:45:25 EDT
I got this error on gluster 3.5.2 on ubuntu 12.04

after extensive use of a dir..

# gluster volume quota sas03 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/EgoTempSata03                           322.0GB       80%     262.4GB  59.6GB

actual usage

# du -sh EgoTempSata03/
81G	EgoTempSata03/

Note You need to log in before you can comment on or make changes to this bug.