Bug 1001556
Summary: | quota: quota list command shows the used size incorrectly on every alternate command execution | ||
---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Rahul Hinduja <rhinduja> |
Component: | glusterd | Assignee: | Pranith Kumar K <pkarampu> |
Status: | CLOSED ERRATA | QA Contact: | Saurabh <saujain> |
Severity: | medium | Docs Contact: | |
Priority: | high | ||
Version: | 2.1 | CC: | kparthas, lmohanty, mzywusko, pkarampu, rhs-bugs, sdharane, spandura, vagarwal, vbellur |
Target Milestone: | --- | Keywords: | ZStream |
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | glusterfs-3.4.0.36rhs-1 | Doc Type: | Bug Fix |
Doc Text: |
Previously, volume quota list command would report different sizes in the "Used" column of the output, on successive runs, when hard limit was reached on a directory. This was seen on plain replicate or distributed replicate type volumes. The quota disk usage can be different in bricks belonging to the same replica set, until the time disk usage accounting converges to reflect the actual disk consumption. In this interval, it was possible to see the difference in volume quota list command as mentioned earlier. Now, with this update, the volume quota list command reports the maximum disk usage, as seen across bricks in a replica set. This make the output consistent.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2013-11-27 15:34:14 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Rahul Hinduja
2013-08-27 09:35:01 UTC
Root cause: Quota size xattrs are not maintained by afr. There is a possibility that they differ even when both the directory changelog xattrs suggest everything is fine. So if there is at least one 'source' check among the sources which has the maximum quota size. Otherwise check among all the available ones for maximum quota size. This way if there is a source and stale copies it always votes for the 'source'. review.gluster.org/6052 is sent upstream to handle this issue for the bug https://bugzilla.redhat.com/show_bug.cgi?id=1002530. While the observed bugs 1002530 and 1001556 and steps to re-create the issues are different, both the issues are happening because the quota size xattrs on the bricks are different. root@pranithk-vm1 - /mnt/r2 17:50:14 :) ⚡ getfattr -d -m. -e hex /brick/r2_10 getfattr: Removing leading '/' from absolute path names # file: brick/r2_10 trusted.afr.r2-client-10=0x000000000000000000000000 trusted.afr.r2-client-11=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x00000000437bf000 trusted.glusterfs.volume-id=0x1fedfd0bd88a4828af6131ef0ef4d641 root@pranith-vm2 - ~ 05:46:59 :( ⚡ getfattr -d -m. -e hex /brick/r2_11 getfattr: Removing leading '/' from absolute path names # file: brick/r2_11 trusted.afr.r2-client-10=0x000000000000000000000000 trusted.afr.r2-client-11=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000043251000 trusted.glusterfs.volume-id=0x1fedfd0bd88a4828af6131ef0ef4d641 Posted the patch https://code.engineering.redhat.com/gerrit/#/c/14129/ downstream. Will move the bug to POST after the patch is sent to u1 branch as well. I have also observed the issue intermittently. This issue still exist. Re-created the issue on build "glusterfs 3.4.0.35rhs built on Oct 15 2013 14:06:04" Steps to recreate the issue: =========================== 1. Create a distribute-replicate volume(4 storage nodes, 3 bricks per node) . Start the volume 2. Create fuse mount. 3. Create 1000 directories from the mount point. 4. killall glusterfsd and glusterfs process on storage_node2 and storage_node4. 5. Enable quota. In a loop set the quota limit-usage of all 1000 directories to 10GB. 6. From mount point create files/directories under each 1000 directories. 7. After a while bring back the bricks. 8. Stop writes from mount point. 9. execute the "quota list on certain directories" from each of the storage node. Output from the quota list command: ================================== +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Source node: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ root@rhs-client13 [Oct-17-2013-12:05:19] >gluster v quota vol_dis_rep list /user2 Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /user2 1.0GB 80% 2.3GB 0Bytes root@rhs-client13 [Oct-17-2013-12:05:05] >getfattr -d -e hex -m . /rhs/bricks/brick7/user2 getfattr: Removing leading '/' from absolute path names # file: rhs/bricks/brick7/user2 trusted.afr.vol_dis_rep-client-6=0x000000000000000000000000 trusted.afr.vol_dis_rep-client-7=0x000000000000000000000000 trusted.gfid=0xb4dba3d0d212441e8a3f1f7c57e56cda trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000019800000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000019800000 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Sink node: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ root@rhs-client14 [Oct-17-2013-12:06:14] >gluster v quota vol_dis_rep list /user2 Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /user2 1.0GB 80% 1.9GB 0Bytes root@rhs-client14 [Oct-17-2013-12:04:59] >getfattr -d -e hex -m . /rhs/bricks/brick8/user2 getfattr: Removing leading '/' from absolute path names # file: rhs/bricks/brick8/user2 trusted.afr.vol_dis_rep-client-6=0x000000000000000000000000 trusted.afr.vol_dis_rep-client-7=0x000000000000000000000000 trusted.gfid=0xb4dba3d0d212441e8a3f1f7c57e56cda trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000000000 Carried the same steps of creating a file of 2GB when the quota limit was set to 1GB. File creation was successful to full 2GB. Recursive listing showed always incremental usage on all the system unlike the issue reported in the bug. Verified with: glusterfs-3.4.0.36rhs-1.el6rhs.x86_64 Output: ======= [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 0Bytes 1.0GB [root@dj ~]# [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 487.0MB 537.0MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 494.5MB 529.5MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 494.5MB 529.5MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 507.3MB 516.7MB [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 535.1MB 488.9MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 535.1MB 488.9MB [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# [root@dj ~]# [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 827.4MB 196.6MB [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1.4GB 0Bytes [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1.4GB 0Bytes [root@dj ~]# [root@dj ~]# [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 2.0GB 0Bytes [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 2.0GB 0Bytes [root@dj ~]# Moving the bug to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html |