Description of problem: ======================= Observed the issue where the disk quota exceeded but the file were created more than the hard limit usage BZ 1001548. Ran the quota list command and it shows the used as 1.3GB->1.4GB->1.3GB->1.4GB alternately, please see the output below for clarification. [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.3GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.4GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.3GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.4GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.3GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.4GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.3GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.4GB 0Bytes [root@nfs1 bricks]# Version-Release number of selected component (if applicable): ============================================================ glusterfs-3.4.0.20rhsquota5-1.el6rhs.x86_64 Steps to Reproduce: =================== 1. Executed quota list command alternatively Actual results: =============== [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.4GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.3GB 0Bytes [root@nfs1 bricks]# gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 1244549331936494413 1.4GB 0Bytes [root@nfs1 bricks]# Additional info: ================ sosreports remains same as of BZ 1001548
Root cause: Quota size xattrs are not maintained by afr. There is a possibility that they differ even when both the directory changelog xattrs suggest everything is fine. So if there is at least one 'source' check among the sources which has the maximum quota size. Otherwise check among all the available ones for maximum quota size. This way if there is a source and stale copies it always votes for the 'source'. review.gluster.org/6052 is sent upstream to handle this issue for the bug https://bugzilla.redhat.com/show_bug.cgi?id=1002530. While the observed bugs 1002530 and 1001556 and steps to re-create the issues are different, both the issues are happening because the quota size xattrs on the bricks are different. root@pranithk-vm1 - /mnt/r2 17:50:14 :) ⚡ getfattr -d -m. -e hex /brick/r2_10 getfattr: Removing leading '/' from absolute path names # file: brick/r2_10 trusted.afr.r2-client-10=0x000000000000000000000000 trusted.afr.r2-client-11=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x00000000437bf000 trusted.glusterfs.volume-id=0x1fedfd0bd88a4828af6131ef0ef4d641 root@pranith-vm2 - ~ 05:46:59 :( ⚡ getfattr -d -m. -e hex /brick/r2_11 getfattr: Removing leading '/' from absolute path names # file: brick/r2_11 trusted.afr.r2-client-10=0x000000000000000000000000 trusted.afr.r2-client-11=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000043251000 trusted.glusterfs.volume-id=0x1fedfd0bd88a4828af6131ef0ef4d641
Posted the patch https://code.engineering.redhat.com/gerrit/#/c/14129/ downstream. Will move the bug to POST after the patch is sent to u1 branch as well.
I have also observed the issue intermittently.
This issue still exist. Re-created the issue on build "glusterfs 3.4.0.35rhs built on Oct 15 2013 14:06:04" Steps to recreate the issue: =========================== 1. Create a distribute-replicate volume(4 storage nodes, 3 bricks per node) . Start the volume 2. Create fuse mount. 3. Create 1000 directories from the mount point. 4. killall glusterfsd and glusterfs process on storage_node2 and storage_node4. 5. Enable quota. In a loop set the quota limit-usage of all 1000 directories to 10GB. 6. From mount point create files/directories under each 1000 directories. 7. After a while bring back the bricks. 8. Stop writes from mount point. 9. execute the "quota list on certain directories" from each of the storage node. Output from the quota list command: ================================== +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Source node: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ root@rhs-client13 [Oct-17-2013-12:05:19] >gluster v quota vol_dis_rep list /user2 Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /user2 1.0GB 80% 2.3GB 0Bytes root@rhs-client13 [Oct-17-2013-12:05:05] >getfattr -d -e hex -m . /rhs/bricks/brick7/user2 getfattr: Removing leading '/' from absolute path names # file: rhs/bricks/brick7/user2 trusted.afr.vol_dis_rep-client-6=0x000000000000000000000000 trusted.afr.vol_dis_rep-client-7=0x000000000000000000000000 trusted.gfid=0xb4dba3d0d212441e8a3f1f7c57e56cda trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000019800000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000019800000 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Sink node: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ root@rhs-client14 [Oct-17-2013-12:06:14] >gluster v quota vol_dis_rep list /user2 Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /user2 1.0GB 80% 1.9GB 0Bytes root@rhs-client14 [Oct-17-2013-12:04:59] >getfattr -d -e hex -m . /rhs/bricks/brick8/user2 getfattr: Removing leading '/' from absolute path names # file: rhs/bricks/brick8/user2 trusted.afr.vol_dis_rep-client-6=0x000000000000000000000000 trusted.afr.vol_dis_rep-client-7=0x000000000000000000000000 trusted.gfid=0xb4dba3d0d212441e8a3f1f7c57e56cda trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000000000
Carried the same steps of creating a file of 2GB when the quota limit was set to 1GB. File creation was successful to full 2GB. Recursive listing showed always incremental usage on all the system unlike the issue reported in the bug. Verified with: glusterfs-3.4.0.36rhs-1.el6rhs.x86_64 Output: ======= [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 0Bytes 1.0GB [root@dj ~]# [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 487.0MB 537.0MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 494.5MB 529.5MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 494.5MB 529.5MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 535.1MB 488.9MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 535.1MB 488.9MB [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# [root@dj ~]# [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1.4GB 0Bytes [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1.4GB 0Bytes [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 2.0GB 0Bytes [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 2.0GB 0Bytes [root@dj ~]# Moving the bug to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html