Bug 989758

Summary: 'gluster volume quota <volname> list' shows inconsistent output
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Dustin Black <dblack>
Component: glusterdAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED ERRATA QA Contact: Saurabh <saujain>
Severity: high Docs Contact:
Priority: high    
Version: 2.1CC: grajaiya, kdhananj, kparthas, mzywusko, pkarampu, rhs-bugs, sasundar, sdharane, shmohan, vagarwal, vbellur, vshastry
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.4.0.34rhs Doc Type: Bug Fix
Doc Text:
Previously, the values in the "Used" and "Available" columns of volume quota list command would fluctuate, even when data is only being written on a quota enabled volume and not removed. Now, with this update, volume quota command output doesn't fluctuate, when data is only being added to the volume.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-27 15:30:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dustin Black 2013-07-29 20:48:46 UTC
Description of problem:
The output of 'gluster volume quota <volname> list' will suddenly fluctuate up and down in "Used" and "Available" values while performing only writes to the volume.

Version-Release number of selected component (if applicable):


How reproducible:
Consistently

Steps to Reproduce:
1. Perform a continuous write action to the volume via an nfs client mount
   $ for i in {1..100}; do sudo dd if=/dev/urandom of=/mnt/test1/file${i} bs=1024k count=100; done
2. Monitor the 'quota list' output at a server node
   # while true; do gluster volume quota rep01 list; sleep 5; done


Actual results:
Output seems to vary in ways not in accordance with the client activity.

Expected results:
Output should be reasonably consistent across RHS nodes and should not move in a direction contrary to the activity on the volume.



Additional info:

Monitoring quota with 'gluster volume quota <volname> list while performing a for loop dd to the volume.

My very well be related to BZ 989753


$ for i in {1..100}; do sudo dd if=/dev/urandom of=/mnt/test1/file${i} bs=1024k count=100; done

# while true; do gluster volume quota rep01 list; sleep 5; done
...
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     299.9MB 724.1MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     352.1MB 671.9MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     343.9MB 680.1MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     849.0MB 175.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     500.0MB 524.0MB
...

Comment 2 Pranith Kumar K 2013-10-10 12:07:30 UTC
Tried re-creating the issue with glusterfs-3.4.0.34rhs. It is working as expected. Here is the log from my testing: The used size is monotonically increasing. So moving to ON_QA for verification on glusterfs-3.4.0.34rhs

                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      0Bytes   1.0GB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      34.0MB 990.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      68.0MB 956.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     101.0MB 923.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     133.0MB 891.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     171.0MB 853.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     207.0MB 817.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     244.0MB 780.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     283.0MB 741.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     311.0MB 713.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     337.0MB 687.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     364.0MB 660.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     402.0MB 622.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     418.0MB 606.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     438.0MB 586.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     463.0MB 561.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     486.0MB 538.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     520.0MB 504.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     556.0MB 468.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     584.0MB 440.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     607.0MB 417.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     633.0MB 391.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     673.0MB 351.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     710.0MB 314.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     742.0MB 282.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     782.0MB 242.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     816.0MB 208.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     851.0MB 173.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     888.0MB 136.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     909.0MB 115.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     942.0MB  82.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     980.0MB  44.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%    1012.0MB  12.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       1.0GB  0Bytes
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       1.0GB  0Bytes
                  Path                   Hard-limit Soft-limit   Used  Available

Comment 3 Pranith Kumar K 2013-10-10 12:09:56 UTC
Forgot to mention that the tests I performed were with both plain replicate and 6x2 distributed replica setups. With both configurations I observed that the functionality is working as expected.

Comment 4 shylesh 2013-10-11 05:15:36 UTC
works fine verified on 3.4.0.34rhs-1.el6rhs.x86_64

                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     920.0MB 104.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     930.0MB  94.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     940.0MB  84.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     940.0MB  84.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     950.0MB  74.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     960.0MB  64.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     960.0MB  64.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     970.0MB  54.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     980.0MB  44.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     990.0MB  34.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     990.0MB  34.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%    1000.0MB  24.0MB

Comment 5 SATHEESARAN 2013-10-15 06:16:01 UTC
After reaching the quota hard-limit, I see that there comes inconsistency in terms of MB/GB.


For instance, look at the console logs below :
I was running, 'quota list' command in loop while running IO on NFS mount.

                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%       1.1GB  0Bytes
                 Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%    1023.9MB  64.0KB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%       1.1GB  0Bytes
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%    1023.9MB  64.0KB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%       1.1GB  0Bytes

Used  space is 1.1 GB in first time and next time its 1023.9MB.
Free space is 0Bytes in first time and next time its 64.0KB

Is this inconsistency to be tolerated ?
Is this a known issue ?

Comment 6 Pranith Kumar K 2013-10-16 05:45:45 UTC
I guess you are talking about this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1001556. The bug is yet to be fixed.

Comment 7 errata-xmlrpc 2013-11-27 15:30:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html