Bug 1001556 - quota: quota list command shows the used size incorrectly on every alternate command execution
quota: quota list command shows the used size incorrectly on every alternate ...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.1
x86_64 Linux
high Severity medium
: ---
: ---
Assigned To: Pranith Kumar K
Saurabh
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-27 05:35 EDT by Rahul Hinduja
Modified: 2016-01-19 01:12 EST (History)
9 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.36rhs-1
Doc Type: Bug Fix
Doc Text:
Previously, volume quota list command would report different sizes in the "Used" column of the output, on successive runs, when hard limit was reached on a directory. This was seen on plain replicate or distributed replicate type volumes. The quota disk usage can be different in bricks belonging to the same replica set, until the time disk usage accounting converges to reflect the actual disk consumption. In this interval, it was possible to see the difference in volume quota list command as mentioned earlier. Now, with this update, the volume quota list command reports the maximum disk usage, as seen across bricks in a replica set. This make the output consistent.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-27 10:34:14 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rahul Hinduja 2013-08-27 05:35:01 EDT
Description of problem:
=======================

Observed the issue where the disk quota exceeded but the file were created more than the hard limit usage BZ 1001548.

Ran the quota list command and it shows the used as 1.3GB->1.4GB->1.3GB->1.4GB alternately, please see the output below for clarification.

[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.3GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.4GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.3GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.4GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.3GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.4GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.3GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.4GB  0Bytes
[root@nfs1 bricks]# 



Version-Release number of selected component (if applicable):
============================================================
glusterfs-3.4.0.20rhsquota5-1.el6rhs.x86_64

Steps to Reproduce:
===================
1. Executed quota list command alternatively 

Actual results:
===============

[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.4GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.3GB  0Bytes
[root@nfs1 bricks]# gluster volume quota dist-rep2 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 1244549331936494413       1.4GB  0Bytes
[root@nfs1 bricks]# 


Additional info:
================

sosreports remains same as of BZ 1001548
Comment 3 Pranith Kumar K 2013-10-10 08:43:38 EDT
Root cause:
Quota size xattrs are not maintained by afr. There is a
possibility that they differ even when both the directory
changelog xattrs suggest everything is fine. So if there is at
least one 'source' check among the sources which has the maximum
quota size. Otherwise check among all the available ones for
maximum quota size. This way if there is a source and stale copies
it always votes for the 'source'.

review.gluster.org/6052 is sent upstream to handle this issue for the bug https://bugzilla.redhat.com/show_bug.cgi?id=1002530.

While the observed bugs 1002530 and 1001556 and steps to re-create the issues are different, both the issues are happening because the quota size xattrs on the bricks are different.

root@pranithk-vm1 - /mnt/r2 
17:50:14 :) ⚡ getfattr -d -m. -e hex /brick/r2_10
getfattr: Removing leading '/' from absolute path names
# file: brick/r2_10
trusted.afr.r2-client-10=0x000000000000000000000000
trusted.afr.r2-client-11=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x00000000437bf000
trusted.glusterfs.volume-id=0x1fedfd0bd88a4828af6131ef0ef4d641


root@pranith-vm2 - ~ 
05:46:59 :( ⚡ getfattr -d -m. -e hex /brick/r2_11
getfattr: Removing leading '/' from absolute path names
# file: brick/r2_11
trusted.afr.r2-client-10=0x000000000000000000000000
trusted.afr.r2-client-11=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000043251000
trusted.glusterfs.volume-id=0x1fedfd0bd88a4828af6131ef0ef4d641
Comment 4 Pranith Kumar K 2013-10-16 04:49:39 EDT
Posted the patch https://code.engineering.redhat.com/gerrit/#/c/14129/ downstream. Will move the bug to POST after the patch is sent to u1 branch as well.
Comment 6 Lalatendu Mohanty 2013-10-17 06:28:31 EDT
I have also observed the issue intermittently.
Comment 7 spandura 2013-10-18 06:52:11 EDT
This issue still exist. Re-created the issue on build "glusterfs 3.4.0.35rhs built on Oct 15 2013 14:06:04" 

Steps to recreate the issue:
===========================
1. Create a distribute-replicate volume(4 storage nodes, 3 bricks per node) . Start the volume

2. Create fuse mount.

3. Create 1000 directories from the mount point.

4. killall glusterfsd and glusterfs process on storage_node2 and storage_node4. 

5. Enable quota. In a loop set the quota limit-usage of all 1000 directories to 10GB. 

6. From mount point create files/directories under each 1000 directories. 

7. After a while bring back the bricks. 

8. Stop writes from mount point. 

9. execute the "quota list on certain directories" from each of the storage node. 

Output from the quota list command:
================================== 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Source node:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
root@rhs-client13 [Oct-17-2013-12:05:19] >gluster v quota vol_dis_rep list /user2
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/user2                                     1.0GB       80%       2.3GB  0Bytes



root@rhs-client13 [Oct-17-2013-12:05:05] >getfattr -d -e hex -m . /rhs/bricks/brick7/user2
getfattr: Removing leading '/' from absolute path names
# file: rhs/bricks/brick7/user2
trusted.afr.vol_dis_rep-client-6=0x000000000000000000000000
trusted.afr.vol_dis_rep-client-7=0x000000000000000000000000
trusted.gfid=0xb4dba3d0d212441e8a3f1f7c57e56cda
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000019800000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000019800000


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Sink node:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

root@rhs-client14 [Oct-17-2013-12:06:14] >gluster v quota vol_dis_rep list /user2
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/user2                                     1.0GB       80%       1.9GB  0Bytes


root@rhs-client14 [Oct-17-2013-12:04:59] >getfattr -d -e hex -m . /rhs/bricks/brick8/user2
getfattr: Removing leading '/' from absolute path names
# file: rhs/bricks/brick8/user2
trusted.afr.vol_dis_rep-client-6=0x000000000000000000000000
trusted.afr.vol_dis_rep-client-7=0x000000000000000000000000
trusted.gfid=0xb4dba3d0d212441e8a3f1f7c57e56cda
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
Comment 8 Rahul Hinduja 2013-10-24 06:54:36 EDT
Carried the same steps of creating a file of 2GB when the quota limit was set to 1GB. File creation was successful to full 2GB.

Recursive listing showed always incremental usage on all the system unlike the issue reported in the bug. 

Verified with: glusterfs-3.4.0.36rhs-1.el6rhs.x86_64

Output:
=======

[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      0Bytes   1.0GB
[root@dj ~]# 
[root@dj ~]# 
[root@dj ~]# 
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     487.0MB 537.0MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     494.5MB 529.5MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     494.5MB 529.5MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     507.3MB 516.7MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     507.3MB 516.7MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     507.3MB 516.7MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     507.3MB 516.7MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     507.3MB 516.7MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     507.3MB 516.7MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     507.3MB 516.7MB
[root@dj ~]# 
[root@dj ~]# 
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     535.1MB 488.9MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     535.1MB 488.9MB
[root@dj ~]# 
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     827.4MB 196.6MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     827.4MB 196.6MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     827.4MB 196.6MB
[root@dj ~]# 
[root@dj ~]# 
[root@dj ~]# 
[root@dj ~]# 
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     827.4MB 196.6MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     827.4MB 196.6MB
[root@dj ~]# 
[root@dj ~]# 
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       1.4GB  0Bytes
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       1.4GB  0Bytes
[root@dj ~]# 
[root@dj ~]# 
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       2.0GB  0Bytes
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       2.0GB  0Bytes

[root@dj ~]# 

Moving the bug to verified.
Comment 9 errata-xmlrpc 2013-11-27 10:34:14 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html

Note You need to log in before you can comment on or make changes to this bug.