Bug 1021466
Summary: | quota: directory limit cross, while creating data in subdirs | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Saurabh <saujain> | |
Component: | quota | Assignee: | Vijaikumar Mallikarjuna <vmallika> | |
Status: | CLOSED WORKSFORME | QA Contact: | storage-qa-internal <storage-qa-internal> | |
Severity: | urgent | Docs Contact: | ||
Priority: | high | |||
Version: | 2.1 | CC: | asriram, gluster-bugs, grajaiya, mhideo, mselvaga, mzywusko, rhs-bugs, rwheeler, smohan, storage-qa-internal, vagarwal, vbellur, vmallika | |
Target Milestone: | --- | Keywords: | ZStream | |
Target Release: | --- | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Known Issue | ||
Doc Text: |
After setting Quota limit on a directory, creating sub directories and populating them with files and renaming the files subsequently while the I/O operation is in progress causes a quota limit violation.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1026291 (view as bug list) | Environment: | ||
Last Closed: | 2015-10-08 07:35:49 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1020127, 1026291 |
Description
Saurabh
2013-10-21 11:06:56 UTC
On a cluster of four nodes, namely, quota[1-4] xattrs for directory in consideration, from node quota1 [root@quota1 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r1/newdir/d4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/d1r1/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x00000000198d0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x00000000198d0000 # file: rhs/brick1/d3r1/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x000000001f3c0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001f3c0000 # file: rhs/brick1/d5r1/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x000000001d640000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001d640000 from node quota2, [root@quota2 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r2/newdir/d4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/d1r2/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x00000000198d0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x00000000198d0000 # file: rhs/brick1/d3r2/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x000000001f3c0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001f3c0000 # file: rhs/brick1/d5r2/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x000000001d640000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001d640000 from node quota3, [root@quota3 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r1/newdir/d4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/d2r1/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x000000001b680000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001b680000 # file: rhs/brick1/d4r1/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x00000000278c0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x00000000278c0000 # file: rhs/brick1/d6r1/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x0000000100000000555555547ffffffd trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x000000001b1a0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001b1a0000 from node quota4, [root@quota4 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r2/newdir/d4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/d2r2/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x000000001b680000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001b680000 # file: rhs/brick1/d4r2/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x00000000278c0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x00000000278c0000 # file: rhs/brick1/d6r2/newdir/d4 trusted.gfid=0x1866f484cc814824a5aa9b1808998ff1 trusted.glusterfs.dht=0x0000000100000000555555547ffffffd trusted.glusterfs.quota.3bc2ab97-9a5b-45d7-b541-c265380d898b.contri=0x000000001b1a0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001b1a0000 after this test, I removed all the files, Tried to create and no renaming this time, Still the limit is getting crossed, [root@quota1 ~]# gluster volume quota dist-rep list /newdir/d4 Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /newdir/d4 1.0GB 80% 2.9GB 0Bytes [root@rhsauto001 newdir]# cd d4 [root@rhsauto001 d4]# cd dir1/dir2/dir3/dir4 [root@rhsauto001 dir4]# du -sh . 2.9G . [root@rhsauto001 dir4]# pwd /mnt/nfs-test/newdir/d4/dir1/dir2/dir3/dir4 [root@rhsauto001 dir4]# mount | grep nfs-test 10.70.42.186:/dist-rep on /mnt/nfs-test type nfs (rw,addr=10.70.42.186) [root@rhsauto001 dir4]# Again going further and tried to test with a new directory "d5" inside the existing parent "newdir" still getting this issue and no rename done meanwhile test is going on, [root@quota1 ~]# gluster volume quota dist-rep list /newdir/d5 Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /newdir/d5 1.0GB 80% 2.8GB 0Bytes [root@rhsauto001 dir4]# pwd /mnt/nfs-test/newdir/d5/dir1/dir2/dir3/dir4 [root@rhsauto001 dir4]# du -sh . 2.9G . [root@rhsauto001 dir4]# Again to check, I exected the test on a new volume and different set of nodes also, similar fashion and no rename involved, issue happens, [root@nfs1 ~]# gluster volume quota dist-rep3 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0TB 80% 3.2GB 1020.8GB /newdir 10.0GB 80% 3.2GB 6.8GB /newdir/d4 1.0GB 80% 3.2GB 0Bytes Volume Name: dist-rep3 Type: Distributed-Replicate Volume ID: 81c83633-0a08-4c27-9ab4-e9aeb1539284 Status: Started Number of Bricks: 6 x 2 = 12 Transport-type: tcp Bricks: Brick1: 10.70.37.213:/rhs/bricks/d1r13 Brick2: 10.70.37.145:/rhs/bricks/d1r23 Brick3: 10.70.37.163:/rhs/bricks/d2r13 Brick4: 10.70.37.76:/rhs/bricks/d2r23 Brick5: 10.70.37.213:/rhs/bricks/d3r13 Brick6: 10.70.37.145:/rhs/bricks/d3r23 Brick7: 10.70.37.163:/rhs/bricks/d4r13 Brick8: 10.70.37.76:/rhs/bricks/d4r23 Brick9: 10.70.37.213:/rhs/bricks/d5r13 Brick10: 10.70.37.145:/rhs/bricks/d5r23 Brick11: 10.70.37.163:/rhs/bricks/d6r13 Brick12: 10.70.37.76:/rhs/bricks/d6r23 Options Reconfigured: features.quota-deem-statfs: on nfs.addr-namelookup: on features.quota: on four node cluster, hostnames nfs[1-4] xattrs from nfs1, [root@nfs1 ~]# getfattr -m . -d -e hex /rhs/bricks/d*r13/newdir/d4 getfattr: Removing leading '/' from absolute path names # file: rhs/bricks/d1r13/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x000000001ee40000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001ee40000 # file: rhs/bricks/d3r13/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x000000002e760000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000002e760000 # file: rhs/bricks/d5r13/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x000000001cd00000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001cd00000 [root@nfs1 ~]# xattrs from nfs2, [root@nfs2 ~]# getfattr -m . -d -e hex /rhs/bricks/d*r23/newdir/d4 getfattr: Removing leading '/' from absolute path names # file: rhs/bricks/d1r23/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x000000001ee40000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001ee40000 # file: rhs/bricks/d3r23/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x000000002e760000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000002e760000 # file: rhs/bricks/d5r23/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x000000001cd00000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001cd00000 xattrs from nfs3, [root@nfs3 ~]# getfattr -m . -d -e hex /rhs/bricks/d*r13/newdir/d4 getfattr: Removing leading '/' from absolute path names # file: rhs/bricks/d2r13/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x0000000023470000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000023470000 # file: rhs/bricks/d4r13/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x0000000020d00000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000020d00000 # file: rhs/bricks/d6r13/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x0000000100000000555555547ffffffd trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x000000001daf0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001daf0000 xattrs from dir4, [root@nfs4 ~]# getfattr -m . -d -e hex /rhs/bricks/d*r23/newdir/d4 getfattr: Removing leading '/' from absolute path names # file: rhs/bricks/d2r23/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x0000000023470000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000023470000 # file: rhs/bricks/d4r23/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x0000000020d00000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000020d00000 # file: rhs/bricks/d6r23/newdir/d4 trusted.gfid=0x564860906ef14367b642a5f810793510 trusted.glusterfs.dht=0x0000000100000000555555547ffffffd trusted.glusterfs.quota.23728d9f-7205-4509-968d-b90d296f4b4f.contri=0x000000001daf0000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x000000001daf0000 Tried out a bit more I am able to cross quota limit by 400%, [root@quota1 ~]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 4.3GB 0Bytes This time the file created in root of the volume only, and the script used to create is similar in comment#0 But, if I use the below mentioned "dd" command, time dd if=/dev/urandom of=f.n bs=102400 count=1024 in a loop, I do not cross the quota limits. as can be seen, [root@quota1 ~]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1.0GB 0Bytes So, this means just by changing the dd params the quota limits are screwed, and we can't keep experimenting all the dd params along with our quota implementation issue can still be seen in glusterfs.3.4.0.36rhs [root@quota1 ~]# gluster volume quota dist-rep3 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 40.0GB 80% 11.0GB 29.0GB /qa1 10.0GB 80% 8.0GB 2.0GB /qa1/dir1-data 1.0GB 80% 1.0GB 0Bytes /qa1/dir2-data 1.0GB 80% 1.5GB 0Bytes /qa1/dir3-data 1.0GB 80% 1.5GB 0Bytes Can you do the same for a size 100GB at the least? (In reply to Vivek Agarwal from comment #8) > Can you do the same for a size 100GB at the least? Yes I have done that, for directories. I have created upto 500GB and "Disk quota exceeded" has occurred. But, if you check the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1024355#c3 The BZ is about addition of quota space used is not done properly. As per this comment I am already telling that we are crossing the limit for "root" of the volume. That puts me in a confusion and makes me think that why we are not aggregating properly and allowing quota limit for root be crossed. Please ignore #c9, as BZ 1024355 has turned out be a NOTABUG. as reply for #c8 Present information is that I have tried out setting limits upto 512GB for a directory and creating data in inside it. The data included creating files sequentially, which has seen "Disk quota exceeded" appropriately The issue is inconsistently reproducible. However with nfs write caching turned off, we never saw this issue even once. So, we suspect that this issue is caused by parallel writes. The following patch make accounting more tight by accounting in-progress (parallel) writes too. https://code.engineering.redhat.com/gerrit/#/c/15126/ Moving the known issues to Doc team, to be documented in release notes for U1 Moving the known issues to Doc team, to be documented in release notes for U1 Moving the known issues to Doc team, to be documented in release notes for U1 I've documented this as a known issue in the Big Bend Update 1 Release Notes. Here is the link: http://documentation-devel.engineering.redhat.com/docs/en-US/Red_Hat_Storage/2.1/html/2.1_Update_1_Release_Notes/chap-Documentation-2.1_Update_1_Release_Notes-Known_Issues.html I tried reproducing it in 3.7.4 gluster v quota vol limit-usage /newdir/d4 512MB 1) gluster v quota vol list /newdir/d4 Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded? ------------------------------------------------------------------------------------------------------------------------------- /newdir/d4 512.0MB 80%(409.6MB) 0Bytes 512.0MB No No 2) Issued in /newdir/d4 -> for i in `seq 1 200`; do dd if=/dev/zero of=f.$i bs=64K count=$i; echo $i; done 3) Meanwhile tried renaming the directory while the IO is happening, mv d4 d5 4) gluster v quota vol list /newdir/d5 Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded? ------------------------------------------------------------------------------------------------------------------------------- /newdir/d5 512.0MB 80%(409.6MB) 514.9MB 0Bytes Yes Yes 5) While performing the IO also, it shows "Disk quota exceeded" 128 dd: failed to open ‘f.129’: Disk quota exceeded 129 dd: failed to open ‘f.130’: Disk quota exceeded 130 dd: failed to open ‘f.131’: Disk quota exceeded 131 dd: failed to open ‘f.132’: Disk quota exceeded 132 dd: failed to open ‘f.133’: Disk quota exceeded 133 dd: failed to open ‘f.134’: Disk quota exceeded 134 dd: failed to open ‘f.135’: Disk quota exceeded |