Bug 765478 - (GLUSTER-3746) mismatch in calculation for quota
mismatch in calculation for quota
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: quota (Show other bugs)
pre-release
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Divya
: Reopened
: 800649 821650 (view as bug list)
Depends On:
Blocks: 839589 848252
  Show dependency treegraph
 
Reported: 2011-10-21 02:12 EDT by Saurabh
Modified: 2015-03-03 04:25 EST (History)
7 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 839589 (view as bug list)
Environment:
Last Closed: 2013-07-24 13:28:24 EDT
Type: ---
Regression: ---
Mount Type: nfs
Documentation: DP
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Saurabh 2011-10-20 23:30:41 EDT
same is the case on fuse mount
Comment 1 Saurabh 2011-10-21 02:12:01 EDT
[root@RHEL6 ~]# glusterfs -V
glusterfs 3.2.5qa2 built on Oct 19 2011 06:23:26
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.



Volume Name: dist-rep1
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.1.11.130:/export/dr1
Brick2: 10.1.11.131:/export/drr1
Brick3: 10.1.11.130:/export/dr2
Brick4: 10.1.11.131:/export/drr2
Options Reconfigured:
features.limit-usage: /:3MB,/d1:1MB,/d2:1MB
features.quota: on


[root@RHEL6 nfs-test]# mount

10.1.11.130:/dist-rep1 on /export/nfs-test type nfs (rw,vers=3,nolock,addr=10.1.11.130)



[root@RHEL6 d1]# ls -l
total 364
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.1
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.10
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.11
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.12
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.13
-rw-r--r--. 1 root root 20480 Oct 21 01:53 f.14
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.15
-rw-r--r--. 1 root root     0 Oct 21 01:53 f.16
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.2
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.3
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.4
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.5
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.6
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.7
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.8
-rw-r--r--. 1 root root 20480 Oct 21 01:52 f.9



[root@RHEL6 d1]# stat f.1
  File: `f.1'
  Size: 20480     	Blocks: 48         IO Block: 65536  regular file
Device: 18h/24d	Inode: 17318786214446460312  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2011-10-21 01:53:18.000000000 -0400
Modify: 2011-10-21 01:52:59.000000000 -0400
Change: 2011-10-21 01:52:59.000000000 -0400



[root@RHEL6 d1]# stat f.16
  File: `f.16'
  Size: 0         	Blocks: 8          IO Block: 65536  regular empty file
Device: 18h/24d	Inode: 17927299769300541861  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2011-10-21 01:53:00.000000000 -0400
Modify: 2011-10-21 01:53:00.000000000 -0400
Change: 2011-10-21 01:53:00.000000000 -0400


As per my calculation the total size should be 364KB but,


[root@RHEL6 ~]# gluster volume quota dist-rep1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           3MB                1.0MB
/d1                         1MB                1.0MB
/d2                         1MB               0Bytes,

extended attributes at this time,

[root@RHEL6 ~]# getfattr -m . -d -e hex /export/dr1/d1/
getfattr: Removing leading '/' from absolute path names
# file: export/dr1/d1/
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xb270c61e6ce9438886a00f9a8f78b63e
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000089000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000000000089000


[root@RHEL6 ~]# getfattr -m . -d -e hex /export/dr2/d1/
getfattr: Removing leading '/' from absolute path names
# file: export/dr2/d1/
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xb270c61e6ce9438886a00f9a8f78b63e
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000077000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000000000077000



Also, after moving all the files from directory "d1" to directory "d2",


[root@RHEL6 ~]# gluster volume quota dist-rep1 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           3MB              428.0KB
/d1                         1MB              364.0KB
/d2                         1MB               64.0KB
Comment 2 Saurabh 2011-10-24 03:25:29 EDT
Adding to this bug,
i have deleted everything from the glusterfs mountpoint

10.1.11.130:/dist-rep2 on /export/gluster-test type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)

still,

[root@RHEL6 ~]# gluster volume quota dist-rep2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           2MB              412.0KB


[root@RHEL6 ~]# getfattr -m . -d -e hex /export/dr-1
getfattr: Removing leading '/' from absolute path names
# file: export/dr-1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dist-rep2-client-0=0x000000000000000000000000
trusted.afr.dist-rep2-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000000000034000
trusted.glusterfs.test=0x776f726b696e6700

[root@RHEL6 ~]# getfattr -m . -d -e hex /export/dr-2
getfattr: Removing leading '/' from absolute path names
# file: export/dr-2
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dist-rep2-client-2=0x000000000000000000000000
trusted.afr.dist-rep2-client-3=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000000000033000
trusted.glusterfs.test=0x776f726b696e6700

[root@RHEL6 ~]# ls /export/dr-1
[root@RHEL6 ~]# ls /export/dr-2
[root@RHEL6 ~]#
Comment 3 Saurabh 2011-10-24 04:32:24 EDT
latest observation,



[root@RHEL6 ~]# gluster volume add-brick dist-rep2 10.1.11.130:/export/add-dr1 10.1.11.131:/export/add-drr1
Add Brick successful
[root@RHEL6 ~]# gluster volume rebalance dist-rep2
Usage: volume rebalance <VOLNAME> [fix-layout|migrate-data] {start|stop|status}
[root@RHEL6 ~]# gluster volume rebalance dist-rep2 start
starting rebalance on volume dist-rep2 has been successful
[root@RHEL6 ~]# gluster volume quota dist-rep2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           2MB                1.1MB
[root@RHEL6 ~]# gluster volume quota dist-rep2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           2MB                1.1MB
[root@RHEL6 ~]# gluster volume quota dist-rep2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           2MB                1.1MB


didn't see this on earlier release
Comment 4 Junaid 2011-10-24 05:23:58 EDT
Executed the same test cases on 3.2qa5 tarball. Not finding the issues. Saurabh please run the tests on tarball.
Comment 5 Saurabh 2011-10-27 03:14:57 EDT
quota calculation mismatch happens because of xfs, as can be found from logs

[root@RHEL6 ~]# mount
/dev/sda3 on / type ext4 (rw)


[root@RHEL6 ~]# gluster volume info dist-rep2

Volume Name: dist-rep2
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.1.11.130:/mnt/dr2
Brick2: 10.1.11.131:/mnt/drr2
Brick3: 10.1.11.130:/mnt/ddr2
Brick4: 10.1.11.131:/mnt/ddrr2
Options Reconfigured:
features.limit-usage: /:2MB
features.quota: on


[root@RHEL6 ~]# gluster volume quota dist-rep2 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           2MB              320.0KB

10.1.11.130:/dist-rep2 on /mnt/glfs type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)

[root@RHEL6 d.0]# ls -l /mnt/glfs
total 16
drwxr-xr-x. 2 root root 8192 Oct 27 06:01 d.0
[root@RHEL6 d.0]# ls -l /mnt/glfs/d.0
total 320
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.1
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.10
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.11
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.12
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.13
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.14
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.15
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.16
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.17
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.18
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.19
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.2
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.20
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.3
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.4
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.5
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.6
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.7
-rw-r--r--. 1 root root 10240 Oct 27 06:01 f.8
-rw-r--r--. 1 root root 10240 Oct 27 06:00 f.9


########################################################

[root@RHEL6 ~]# mount
/dev/sdb1 on /export type xfs (rw)


[root@RHEL6 ~]# gluster volume info dist-rep3

Volume Name: dist-rep3
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.1.11.130:/export/dr3
Brick2: 10.1.11.131:/export/drr3
Brick3: 10.1.11.130:/export/ddr3
Brick4: 10.1.11.131:/export/ddrr3
Options Reconfigured:
features.limit-usage: /:2MB
features.quota: on
[root@RHEL6 ~]# gluster volume quota dist-rep3 list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                           2MB                1.3MB


10.1.11.130:/dist-rep3 on /mnt/glfs1 type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)


[root@RHEL6 d.0]# ls -l /mnt/glfs1
total 12
drwxr-xr-x. 2 root root 4197 Oct 27 05:47 d.0
[root@RHEL6 d.0]# ls -l /mnt/glfs1/d.0
total 320
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.1
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.10
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.11
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.12
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.13
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.14
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.15
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.16
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.17
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.18
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.19
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.2
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.20
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.3
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.4
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.5
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.6
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.7
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.8
-rw-r--r--. 1 root root 10240 Oct 27 05:47 f.9
Comment 6 Junaid 2011-11-03 03:33:45 EDT
Saurabh,

XFS by default uses 128 blocs as io-buffers for doing contiguous writes. So, even though we write a byte to the file on XFS it is allocated 128 blocks during the life of the files. Once the file is closed the left over blocks are freed. Hence, when quota is doing file calculation while the file is open it will see 128 * 512 bytes and marks it as the size of the file.

To avoid this we must mount the XFS file system with the option allocsize=4096. This is avoid this problem.

mount -t xfs -o allocsize=4096 device mount-point

We must document this.
Comment 7 Saurabh 2011-11-03 06:51:53 EDT
with the workaround it is working, just running the regular sanity tests and then plan to run the quota-tests over this arrangement.
Comment 8 Vidya Sakar 2012-05-31 04:51:31 EDT
Let us move the bug to verified after we document this in our admin guide.
Comment 9 Junaid 2012-06-13 04:28:54 EDT
*** Bug 800649 has been marked as a duplicate of this bug. ***
Comment 11 Vijaikumar Mallikarjuna 2015-03-03 04:25:39 EST
*** Bug 821650 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.