Description of problem: The output of 'gluster volume quota <volname> list' will suddenly fluctuate up and down in "Used" and "Available" values while performing only writes to the volume. Version-Release number of selected component (if applicable): How reproducible: Consistently Steps to Reproduce: 1. Perform a continuous write action to the volume via an nfs client mount $ for i in {1..100}; do sudo dd if=/dev/urandom of=/mnt/test1/file${i} bs=1024k count=100; done 2. Monitor the 'quota list' output at a server node # while true; do gluster volume quota rep01 list; sleep 5; done Actual results: Output seems to vary in ways not in accordance with the client activity. Expected results: Output should be reasonably consistent across RHS nodes and should not move in a direction contrary to the activity on the volume. Additional info: Monitoring quota with 'gluster volume quota <volname> list while performing a for loop dd to the volume. My very well be related to BZ 989753 $ for i in {1..100}; do sudo dd if=/dev/urandom of=/mnt/test1/file${i} bs=1024k count=100; done # while true; do gluster volume quota rep01 list; sleep 5; done ... Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1GB 90% 299.9MB 724.1MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1GB 90% 352.1MB 671.9MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1GB 90% 343.9MB 680.1MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1GB 90% 849.0MB 175.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1GB 90% 500.0MB 524.0MB ...
Tried re-creating the issue with glusterfs-3.4.0.34rhs. It is working as expected. Here is the log from my testing: The used size is monotonically increasing. So moving to ON_QA for verification on glusterfs-3.4.0.34rhs Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 0Bytes 1.0GB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 34.0MB 990.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 68.0MB 956.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 101.0MB 923.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 133.0MB 891.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 171.0MB 853.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 207.0MB 817.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 244.0MB 780.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 283.0MB 741.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 311.0MB 713.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 337.0MB 687.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 364.0MB 660.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 402.0MB 622.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 418.0MB 606.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 438.0MB 586.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 463.0MB 561.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 486.0MB 538.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 520.0MB 504.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 556.0MB 468.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 584.0MB 440.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 607.0MB 417.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 633.0MB 391.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 673.0MB 351.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 710.0MB 314.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 742.0MB 282.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 782.0MB 242.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 816.0MB 208.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 851.0MB 173.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 888.0MB 136.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 909.0MB 115.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 942.0MB 82.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 980.0MB 44.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1012.0MB 12.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1.0GB 0Bytes Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1.0GB 0Bytes Path Hard-limit Soft-limit Used Available
Forgot to mention that the tests I performed were with both plain replicate and 6x2 distributed replica setups. With both configurations I observed that the functionality is working as expected.
works fine verified on 3.4.0.34rhs-1.el6rhs.x86_64 Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 920.0MB 104.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 930.0MB 94.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 940.0MB 84.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 940.0MB 84.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 950.0MB 74.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 960.0MB 64.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 960.0MB 64.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 970.0MB 54.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 980.0MB 44.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 990.0MB 34.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 990.0MB 34.0MB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1000.0MB 24.0MB
After reaching the quota hard-limit, I see that there comes inconsistency in terms of MB/GB. For instance, look at the console logs below : I was running, 'quota list' command in loop while running IO on NFS mount. Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /master 1.0GB 80% 1.1GB 0Bytes Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /master 1.0GB 80% 1023.9MB 64.0KB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /master 1.0GB 80% 1.1GB 0Bytes Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /master 1.0GB 80% 1023.9MB 64.0KB Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /master 1.0GB 80% 1.1GB 0Bytes Used space is 1.1 GB in first time and next time its 1023.9MB. Free space is 0Bytes in first time and next time its 64.0KB Is this inconsistency to be tolerated ? Is this a known issue ?
I guess you are talking about this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1001556. The bug is yet to be fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html