Bug 989758 - 'gluster volume quota <volname> list' shows inconsistent output
'gluster volume quota <volname> list' shows inconsistent output
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.1
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Pranith Kumar K
Saurabh
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-29 16:48 EDT by Dustin Black
Modified: 2016-01-19 01:12 EST (History)
12 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.34rhs
Doc Type: Bug Fix
Doc Text:
Previously, the values in the "Used" and "Available" columns of volume quota list command would fluctuate, even when data is only being written on a quota enabled volume and not removed. Now, with this update, volume quota command output doesn't fluctuate, when data is only being added to the volume.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-27 10:30:42 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dustin Black 2013-07-29 16:48:46 EDT
Description of problem:
The output of 'gluster volume quota <volname> list' will suddenly fluctuate up and down in "Used" and "Available" values while performing only writes to the volume.

Version-Release number of selected component (if applicable):


How reproducible:
Consistently

Steps to Reproduce:
1. Perform a continuous write action to the volume via an nfs client mount
   $ for i in {1..100}; do sudo dd if=/dev/urandom of=/mnt/test1/file${i} bs=1024k count=100; done
2. Monitor the 'quota list' output at a server node
   # while true; do gluster volume quota rep01 list; sleep 5; done


Actual results:
Output seems to vary in ways not in accordance with the client activity.

Expected results:
Output should be reasonably consistent across RHS nodes and should not move in a direction contrary to the activity on the volume.



Additional info:

Monitoring quota with 'gluster volume quota <volname> list while performing a for loop dd to the volume.

My very well be related to BZ 989753


$ for i in {1..100}; do sudo dd if=/dev/urandom of=/mnt/test1/file${i} bs=1024k count=100; done

# while true; do gluster volume quota rep01 list; sleep 5; done
...
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     299.9MB 724.1MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     352.1MB 671.9MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     343.9MB 680.1MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     849.0MB 175.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                            1GB       90%     500.0MB 524.0MB
...
Comment 2 Pranith Kumar K 2013-10-10 08:07:30 EDT
Tried re-creating the issue with glusterfs-3.4.0.34rhs. It is working as expected. Here is the log from my testing: The used size is monotonically increasing. So moving to ON_QA for verification on glusterfs-3.4.0.34rhs

                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      0Bytes   1.0GB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      34.0MB 990.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      68.0MB 956.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     101.0MB 923.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     133.0MB 891.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     171.0MB 853.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     207.0MB 817.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     244.0MB 780.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     283.0MB 741.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     311.0MB 713.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     337.0MB 687.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     364.0MB 660.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     402.0MB 622.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     418.0MB 606.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     438.0MB 586.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     463.0MB 561.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     486.0MB 538.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     520.0MB 504.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     556.0MB 468.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     584.0MB 440.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     607.0MB 417.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     633.0MB 391.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     673.0MB 351.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     710.0MB 314.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     742.0MB 282.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     782.0MB 242.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     816.0MB 208.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     851.0MB 173.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     888.0MB 136.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     909.0MB 115.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     942.0MB  82.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     980.0MB  44.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%    1012.0MB  12.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       1.0GB  0Bytes
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       1.0GB  0Bytes
                  Path                   Hard-limit Soft-limit   Used  Available
Comment 3 Pranith Kumar K 2013-10-10 08:09:56 EDT
Forgot to mention that the tests I performed were with both plain replicate and 6x2 distributed replica setups. With both configurations I observed that the functionality is working as expected.
Comment 4 shylesh 2013-10-11 01:15:36 EDT
works fine verified on 3.4.0.34rhs-1.el6rhs.x86_64

                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     920.0MB 104.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     930.0MB  94.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     940.0MB  84.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     940.0MB  84.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     950.0MB  74.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     960.0MB  64.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     960.0MB  64.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     970.0MB  54.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     980.0MB  44.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     990.0MB  34.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     990.0MB  34.0MB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%    1000.0MB  24.0MB
Comment 5 SATHEESARAN 2013-10-15 02:16:01 EDT
After reaching the quota hard-limit, I see that there comes inconsistency in terms of MB/GB.


For instance, look at the console logs below :
I was running, 'quota list' command in loop while running IO on NFS mount.

                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%       1.1GB  0Bytes
                 Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%    1023.9MB  64.0KB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%       1.1GB  0Bytes
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%    1023.9MB  64.0KB
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/master                                    1.0GB       80%       1.1GB  0Bytes

Used  space is 1.1 GB in first time and next time its 1023.9MB.
Free space is 0Bytes in first time and next time its 64.0KB

Is this inconsistency to be tolerated ?
Is this a known issue ?
Comment 6 Pranith Kumar K 2013-10-16 01:45:45 EDT
I guess you are talking about this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1001556. The bug is yet to be fixed.
Comment 7 errata-xmlrpc 2013-11-27 10:30:42 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html

Note You need to log in before you can comment on or make changes to this bug.