Bug 1023134

Summary: Used disk size reported by quota and du mismatch
Product: [Community] GlusterFS Reporter: Paschalis Korosoglou <pkoro>
Component: quotaAssignee: Vijaikumar Mallikarjuna <vmallika>
Status: CLOSED EOL QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: bugs, gluster-bugs, pauyeung, smohan, sysadmin, vmallika
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-07 12:13:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Quota Verification Tool none

Description Paschalis Korosoglou 2013-10-24 17:09:22 UTC 
Description of problem:

Used disk size reported by quota and du mismatch. 

The following commands (executed on the server side) showcase the problem. The result of du command on the clients tested reports the same used size as on the server. 

# gluster volume status uiwork
Status of volume: uiwork
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick gluster05.gluster:/mnt/work01/brick		49164	Y	2759
Brick gluster01.gluster:/mnt/work01/brick		49160	Y	11252
Self-heal Daemon on localhost				N/A	Y	59288
Self-heal Daemon on gluster03.gluster			N/A	Y	25472
Self-heal Daemon on gluster02.gluster			N/A	Y	25583
Self-heal Daemon on gluster01.gluster			N/A	Y	26455
 
There are no active volume tasks
# du -s --si /mnt/work01/brick/home/johndoe/
41G	/mnt/work01/brick/home/johndoe/
# gluster volume quota uiwork list /home/johndoe 
	path		  limit_set	     size
----------------------------------------------------------------------------------
/home/johndoe             60GB               52.6GB

We have tried removing and re-adding the limit for this path, but each time gluster reports a different value from what seems to be the actual one. 

I have checked for hard links but there seem to be none on the client side. In general I have tried cross checking this with other tickets where a similar problem is expressed but couldn't find something in clear relation. 

The underlying fs on the bricks is xfs. 

Please let me know of any further information I need to send you on this issue. 

Version-Release number of selected component (if applicable):

On gluster server: 3.4.0-2.el6
On gluster clients: 3.4.0-2.el5 and 3.4.0-8.el6

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Peter Auyeung 2014-08-29 17:51:00 UTC 
I got into this bug with NFS.
I am running gluster 3.5.2 on ubuntu 12.04 xfs bricks

# gluster volume quota sas03 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/EgoTempSata03                           322.0GB       80%     262.4GB  59.6GB

# du -sh EgoTempSata03/
81G	EgoTempSata03/
Comment 2 NOVELTIS IT 2014-11-25 08:18:38 UTC 
I am experiencing exactly the same issue with Gluster 3.5.3 (Debian Wheezy package release).

gluster volume quota projets list
/7028-OPERA                               75.0GB       80%      77.2GB  0Bytes

# du -sh 7028-OPERA
67G     /gluster-projets/7028-OPERA
Comment 3 Peter Auyeung 2014-12-23 19:09:50 UTC 
I have features.quota-deem-statfs on and no hardlinks on the volume.

I also moved the date/volume from a replicated filesystem to a distributed one and still having the mismatch
Comment 4 Vijaikumar Mallikarjuna 2015-01-22 07:47:04 UTC 
Created attachment 982627 [details]
Quota Verification Tool

Hi Peter,

Can you please run the attached script 'quota-verify' in the back-end for all the bricks where quota is enabled and provide us the output?

The script basically crawls the directory given as argument.
It collects quota "contri" and "size" extended attribute and
also "block size" from stat call. script can create log whose size can be few MBs

This script needs to be executed individually for all the bricks.


Thanks and Regards,
Vijay
Comment 5 Niels de Vos 2015-05-17 21:59:56 UTC 
GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5.

This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "bugs".

If there is no response by the end of the month, this bug will get automatically closed.
Comment 6 Kaleb KEITHLEY 2015-10-07 12:13:34 UTC 
GlusterFS 3.4.x has reached end-of-life.

If this bug still exists in a later release please reopen this and change the version or open a new bug.