Bug 917901

Summary:

Mismatch in calculation for quota directory

Product:

[Community] GlusterFS

Reporter:

Ramesh <ramesh.kumar2>

Component:

quota

Assignee:

Vijaikumar Mallikarjuna <vmallika>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Severity:

urgent

Docs Contact:

Priority:

high

Version:

3.5.2

CC:

bugs, c.affolter, kaushal, miteshpathak05, pauyeung, smohan, vmallika

Target Milestone:

---

Keywords:

Triaged

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-04-07 01:35:38 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Quota Verification Tool	none

Description Ramesh 2013-03-05 04:48:30 UTC

Description of problem:

Glusterfs quota directory showing incorrect useable size. I have enable quota on a directory and it is showing incorrect size of that directory. I have defined 1.1 TB for a directory and actual used size of that directory is 600GB when I check using du command. but when i am checking using gluster volume quota command then it is showing 800GB so there is difference of 200GB.  

Version-Release number of selected component (if applicable):
[root@akash ~]# glusterfs --version
glusterfs 3.2.7 built on Nov 27 2012 15:03:22
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

Comment 1 Mitesh Pathak [:mr_pathak] 2013-03-10 06:59:57 UTC

Hi Folks 
I am new to GFS, and would love to start working on bugs.
Is this a valid bug & available to work on ?

Regards,
Mits

Comment 2 Ramesh 2013-03-10 07:15:53 UTC

Hi, 

I am still waiting for response to proceed this bug.

Please do the needful. 



Regards,
Ramesh

Comment 3 vpshastry 2013-03-11 06:15:37 UTC

Hi Ramesh && Mitesh,
There is a similar bug reported, and it might be a duplicate of the same https://bugzilla.redhat.com/show_bug.cgi?id=812230.

This happens in 2 cases,
* If you've created hard links in different directories, quota treats it as 2 different contributions, where as du (disk usage) is not doubled for creating hard links.
* Quota stores the limit in inode_ctx, and lookup (see quota_lookup_cbk) is responsible for updating the ctx. But in NFS mount the NFS doesn't send lookups, leading to wrong calculations of quota limits.

If your problem relates to 1st case, its an expected behaviour, else its a bug as in later mentioned case.

Wish you happy hacking into GFS.

Comment 5 Peter Auyeung 2014-08-29 06:47:07 UTC

I got into this bug with NFS.
I am running gluster 3.5.2 on ubuntu 12.04 xfs bricks

# gluster volume quota sas03 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/EgoTempSata03                           322.0GB       80%     262.4GB  59.6GB

# du -sh EgoTempSata03/
81G	EgoTempSata03/

Comment 6 Niels de Vos 2014-11-11 09:37:15 UTC

Hi Peter,

could you let us know if there are hard-links in the contents stored on your volume/directory?

Thanks,
Niels

Comment 7 Peter Auyeung 2014-12-23 19:07:06 UTC

There are no hard link on the volume/dir and i already have features.quota-deem-statfs on

Comment 8 Peter Auyeung 2014-12-23 19:08:15 UTC

I moved the volume from a replicated to a pure distributed volume and the mismatch still happening

Comment 9 Vijaikumar Mallikarjuna 2015-01-22 07:47:41 UTC

Created attachment 982628 [details]
Quota Verification Tool

Hi Peter,

Can you please run the attached script 'quota-verify' in the back-end for all the bricks where quota is enabled and provide us the output?

The script basically crawls the directory given as argument.
It collects quota "contri" and "size" extended attribute and
also "block size" from stat call. script can create log whose size can be few MBs

This script needs to be executed individually for all the bricks.


Thanks and Regards,
Vijay

Comment 10 Peter Auyeung 2015-01-22 20:45:11 UTC

Hi Vijaikumar

Here are the outputs of all our 6 nodes on a particular brick path.
It clearly shows mismatch

Thanks!
Peter

root@glusterprod001:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.2	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.4	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.12	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.26	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.36	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.41	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.45	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.47	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.48	 = 1999867904 (links: 1)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02/iqqa01	 = 229984161280
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (229984161280 != 17998811136)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02	 = 229984161280
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (229984161280 != 17998811136)

root@glusterprod002:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.5	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.6	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.8	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.9	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.18	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.24	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.31	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.42	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.49	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.60	 = 1999867904 (links: 1)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02/iqqa01	 = 199697740288
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (199697740288 != 19998679040)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02	 = 199697740288
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (199697740288 != 19998679040)

root@glusterprod003:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.1	 = 1999220224 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.13	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.16	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.20	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.30	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.50	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.58	 = 1999867904 (links: 1)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02/iqqa01	 = 115703288320
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (115703288320 != 13998427648)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02	 = 115703288320
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (115703288320 != 13998427648)

root@glusterprod004:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.14	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.21	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.27	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.32	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.35	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.39	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.40	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.55	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.61	 = 1999867904 (links: 1)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02/iqqa01	 = 143990489088
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (143990489088 != 17998811136)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02	 = 143990489088
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (143990489088 != 17998811136)

root@glusterprod005:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.3	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.22	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.25	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.28	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.33	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.34	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.43	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.44	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.51	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.53	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.54	 = 1999867904 (links: 1)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02/iqqa01	 = 139844609024
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (139844609024 != 21998546944)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02	 = 139844609024
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (139844609024 != 21998546944)

root@glusterprod006:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.7	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.10	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.11	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.15	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.17	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.19	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.23	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.29	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.37	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.38	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.46	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.52	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.56	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.57	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.59	 = 1999867904 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.62	 = 1855666688 (links: 1)
[DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa01_date.txt	 = 512 (links: 1)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02/iqqa01	 = 159555537408
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (159555537408 != 31853685760)
Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120.
[DEBUG] dir  size   /brick04/gfs/AseIqBackupSata02	 = 159555537408
[ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (159555537408 != 31853685760)

Comment 11 Vijaikumar Mallikarjuna 2015-01-23 06:32:25 UTC

Hi Peter,

Try below workaround and see if this helps solving the problem

Set the below mentioned directory as dirty from back-end on all the nodes in cluster:

# setfattr -n trusted.glusterfs.quota.dirty -v 0x3100 /brick04/gfs/AseIqBackupSata02/iqqa01

# setfattr -n trusted.glusterfs.quota.dirty -v 0x3100 /brick04/gfs/AseIqBackupSata02



From mount point initiate readdirp
# ls -l /iqqa01
# ls -l /

Comment 12 Peter Auyeung 2015-01-23 20:17:19 UTC

Tested on DEV system and works!

However when we did that on prod, it brought down the ALL the bricks on that volume.

I have to start force for that volume and at the end it fixed the quota

-Peter

Comment 13 Peter Auyeung 2015-01-23 20:17:57 UTC

is there any less intrusive way to do this?

Seems like the number of files might affect the quota rescan process?

Comment 14 Vijaikumar Mallikarjuna 2015-01-27 06:20:56 UTC

Hi Peter,

Brick going down may be because of bug# 1176393 with readdirp, this is now fix in latest code.


With the re-scan process, the only impact could be performance (depends on the depth of the sub-directory) tree till the process is complete.

We are already working on improving quota accounting, this will solve the accounting going wrong.

Comment 15 Peter Auyeung 2015-01-27 22:30:12 UTC

Does lastest code means 3.6.2 or 3.5.3?

I am on 3.5.2 and hope same version patch would fix this.

Thanks
Peter

Comment 16 Vijaikumar Mallikarjuna 2015-01-28 05:41:53 UTC

Hi Peter,

Bug# 1176393 is fixed in 3.7. We will back-port the same to 3.6

Comment 17 Peter Auyeung 2015-01-28 05:44:21 UTC

How about back port to 3.5??

Thanks!
Peter

Comment 18 Vijaikumar Mallikarjuna 2015-01-28 06:38:48 UTC

Sure, we will back-port the fix to 3.5 and 3.6

Comment 19 Christian Affolter 2016-02-24 15:05:55 UTC

(In reply to Vijaikumar Mallikarjuna from comment #18)
> Sure, we will back-port the fix to 3.5 and 3.6

Is it still planned to back-port this to 3.5?

Thanks!
Chris

Comment 20 Vijaikumar Mallikarjuna 2016-02-25 05:32:55 UTC

Hi Christian,

In 3.7, we have made multiple fixes and quota-marker code is re-written.
So it is hard to back-port these changes to 3.5

Thanks,
Vijay

Comment 21 Vijaikumar Mallikarjuna 2016-04-07 01:35:38 UTC

Many issues related to quota is fixed in 3.7.11, please file a new bug if it happens again in 3.7.11 or greater