Bug 765478 (GLUSTER-3746)
| Summary: | mismatch in calculation for quota | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Saurabh <saurabh> | |
| Component: | quota | Assignee: | Divya <divya> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | pre-release | CC: | amarts, amaya, gluster-bugs, junaid, rfortier, saujain, shmohan | |
| Target Milestone: | --- | Keywords: | Reopened | |
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.4.0 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 839589 (view as bug list) | Environment: | ||
| Last Closed: | 2013-07-24 17:28:24 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | nfs | |
| Documentation: | DP | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 839589, 848252 | |||
|
Description
Saurabh
2011-10-21 03:30:41 UTC
[root@RHEL6 ~]# glusterfs -V glusterfs 3.2.5qa2 built on Oct 19 2011 06:23:26 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. Volume Name: dist-rep1 Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.1.11.130:/export/dr1 Brick2: 10.1.11.131:/export/drr1 Brick3: 10.1.11.130:/export/dr2 Brick4: 10.1.11.131:/export/drr2 Options Reconfigured: features.limit-usage: /:3MB,/d1:1MB,/d2:1MB features.quota: on [root@RHEL6 nfs-test]# mount 10.1.11.130:/dist-rep1 on /export/nfs-test type nfs (rw,vers=3,nolock,addr=10.1.11.130) [root@RHEL6 d1]# ls -l total 364 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.1 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.10 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.11 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.12 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.13 -rw-r--r--. 1 root root 20480 Oct 21 01:53 f.14 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.15 -rw-r--r--. 1 root root 0 Oct 21 01:53 f.16 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.2 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.3 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.4 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.5 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.6 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.7 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.8 -rw-r--r--. 1 root root 20480 Oct 21 01:52 f.9 [root@RHEL6 d1]# stat f.1 File: `f.1' Size: 20480 Blocks: 48 IO Block: 65536 regular file Device: 18h/24d Inode: 17318786214446460312 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-10-21 01:53:18.000000000 -0400 Modify: 2011-10-21 01:52:59.000000000 -0400 Change: 2011-10-21 01:52:59.000000000 -0400 [root@RHEL6 d1]# stat f.16 File: `f.16' Size: 0 Blocks: 8 IO Block: 65536 regular empty file Device: 18h/24d Inode: 17927299769300541861 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2011-10-21 01:53:00.000000000 -0400 Modify: 2011-10-21 01:53:00.000000000 -0400 Change: 2011-10-21 01:53:00.000000000 -0400 As per my calculation the total size should be 364KB but, [root@RHEL6 ~]# gluster volume quota dist-rep1 list path limit_set size ---------------------------------------------------------------------------------- / 3MB 1.0MB /d1 1MB 1.0MB /d2 1MB 0Bytes, extended attributes at this time, [root@RHEL6 ~]# getfattr -m . -d -e hex /export/dr1/d1/ getfattr: Removing leading '/' from absolute path names # file: export/dr1/d1/ security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xb270c61e6ce9438886a00f9a8f78b63e trusted.glusterfs.dht=0x0000000100000000000000007ffffffe trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000089000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x0000000000089000 [root@RHEL6 ~]# getfattr -m . -d -e hex /export/dr2/d1/ getfattr: Removing leading '/' from absolute path names # file: export/dr2/d1/ security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xb270c61e6ce9438886a00f9a8f78b63e trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000077000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x0000000000077000 Also, after moving all the files from directory "d1" to directory "d2", [root@RHEL6 ~]# gluster volume quota dist-rep1 list path limit_set size ---------------------------------------------------------------------------------- / 3MB 428.0KB /d1 1MB 364.0KB /d2 1MB 64.0KB Adding to this bug, i have deleted everything from the glusterfs mountpoint 10.1.11.130:/dist-rep2 on /export/gluster-test type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072) still, [root@RHEL6 ~]# gluster volume quota dist-rep2 list path limit_set size ---------------------------------------------------------------------------------- / 2MB 412.0KB [root@RHEL6 ~]# getfattr -m . -d -e hex /export/dr-1 getfattr: Removing leading '/' from absolute path names # file: export/dr-1 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dist-rep2-client-0=0x000000000000000000000000 trusted.afr.dist-rep2-client-1=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000007ffffffe trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x0000000000034000 trusted.glusterfs.test=0x776f726b696e6700 [root@RHEL6 ~]# getfattr -m . -d -e hex /export/dr-2 getfattr: Removing leading '/' from absolute path names # file: export/dr-2 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dist-rep2-client-2=0x000000000000000000000000 trusted.afr.dist-rep2-client-3=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x0000000000033000 trusted.glusterfs.test=0x776f726b696e6700 [root@RHEL6 ~]# ls /export/dr-1 [root@RHEL6 ~]# ls /export/dr-2 [root@RHEL6 ~]# latest observation,
[root@RHEL6 ~]# gluster volume add-brick dist-rep2 10.1.11.130:/export/add-dr1 10.1.11.131:/export/add-drr1
Add Brick successful
[root@RHEL6 ~]# gluster volume rebalance dist-rep2
Usage: volume rebalance <VOLNAME> [fix-layout|migrate-data] {start|stop|status}
[root@RHEL6 ~]# gluster volume rebalance dist-rep2 start
starting rebalance on volume dist-rep2 has been successful
[root@RHEL6 ~]# gluster volume quota dist-rep2 list
path limit_set size
----------------------------------------------------------------------------------
/ 2MB 1.1MB
[root@RHEL6 ~]# gluster volume quota dist-rep2 list
path limit_set size
----------------------------------------------------------------------------------
/ 2MB 1.1MB
[root@RHEL6 ~]# gluster volume quota dist-rep2 list
path limit_set size
----------------------------------------------------------------------------------
/ 2MB 1.1MB
didn't see this on earlier release
Executed the same test cases on 3.2qa5 tarball. Not finding the issues. Saurabh please run the tests on tarball. quota calculation mismatch happens because of xfs, as can be found from logs [root@RHEL6 ~]# mount /dev/sda3 on / type ext4 (rw) [root@RHEL6 ~]# gluster volume info dist-rep2 Volume Name: dist-rep2 Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.1.11.130:/mnt/dr2 Brick2: 10.1.11.131:/mnt/drr2 Brick3: 10.1.11.130:/mnt/ddr2 Brick4: 10.1.11.131:/mnt/ddrr2 Options Reconfigured: features.limit-usage: /:2MB features.quota: on [root@RHEL6 ~]# gluster volume quota dist-rep2 list path limit_set size ---------------------------------------------------------------------------------- / 2MB 320.0KB 10.1.11.130:/dist-rep2 on /mnt/glfs type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072) [root@RHEL6 d.0]# ls -l /mnt/glfs total 16 drwxr-xr-x. 2 root root 8192 Oct 27 06:01 d.0 [root@RHEL6 d.0]# ls -l /mnt/glfs/d.0 total 320 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.1 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.10 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.11 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.12 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.13 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.14 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.15 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.16 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.17 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.18 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.19 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.2 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.20 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.3 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.4 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.5 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.6 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.7 -rw-r--r--. 1 root root 10240 Oct 27 06:01 f.8 -rw-r--r--. 1 root root 10240 Oct 27 06:00 f.9 ######################################################## [root@RHEL6 ~]# mount /dev/sdb1 on /export type xfs (rw) [root@RHEL6 ~]# gluster volume info dist-rep3 Volume Name: dist-rep3 Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.1.11.130:/export/dr3 Brick2: 10.1.11.131:/export/drr3 Brick3: 10.1.11.130:/export/ddr3 Brick4: 10.1.11.131:/export/ddrr3 Options Reconfigured: features.limit-usage: /:2MB features.quota: on [root@RHEL6 ~]# gluster volume quota dist-rep3 list path limit_set size ---------------------------------------------------------------------------------- / 2MB 1.3MB 10.1.11.130:/dist-rep3 on /mnt/glfs1 type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072) [root@RHEL6 d.0]# ls -l /mnt/glfs1 total 12 drwxr-xr-x. 2 root root 4197 Oct 27 05:47 d.0 [root@RHEL6 d.0]# ls -l /mnt/glfs1/d.0 total 320 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.1 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.10 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.11 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.12 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.13 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.14 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.15 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.16 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.17 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.18 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.19 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.2 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.20 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.3 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.4 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.5 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.6 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.7 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.8 -rw-r--r--. 1 root root 10240 Oct 27 05:47 f.9 Saurabh, XFS by default uses 128 blocs as io-buffers for doing contiguous writes. So, even though we write a byte to the file on XFS it is allocated 128 blocks during the life of the files. Once the file is closed the left over blocks are freed. Hence, when quota is doing file calculation while the file is open it will see 128 * 512 bytes and marks it as the size of the file. To avoid this we must mount the XFS file system with the option allocsize=4096. This is avoid this problem. mount -t xfs -o allocsize=4096 device mount-point We must document this. with the workaround it is working, just running the regular sanity tests and then plan to run the quota-tests over this arrangement. Let us move the bug to verified after we document this in our admin guide. *** Bug 800649 has been marked as a duplicate of this bug. *** Updated documentation is available at: http://documentation-stage.bne.redhat.com/docs/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/chap-User_Guide-Setting_Volumes.html#id3088942 *** Bug 821650 has been marked as a duplicate of this bug. *** |