Bug 1399066

Summary:	Quota is not listing proper information and leading to disc full status
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Vivek Das <vdas>
Component:	quota	Assignee:	hari gowtham <hgowtham>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.2	CC:	amukherj, atoborek, bkunal, hgowtham, olim, rhs-bugs, storage-qa-internal, vdas
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	Accounting
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1421933 (view as bug list)		Environment:
Last Closed:	2018-12-28 11:06:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1421933
Bug Blocks:

Description Vivek Das 2016-11-28 08:39:04 UTC

Description of problem:
On a four node samba ctdb setup enable quota and set the quota limit. When we do a windows mount it shows disc size full and does not allow to right any data to it where as the share is completely empty without any data present in it.
gluster volume quota volname list is showing used space as 16384.0PB.
Even on cifs mount right is not possible because of the disc size full issue.

Version-Release number of selected component (if applicable):
samba-4.4.6-2.el7rhgs.x86_64
glusterfs-fuse-3.8.4-5.el7rhgs.x86_64

How reproducible:
1/1

Steps to Reproduce:
1.Enable quota on a samba-ctdb setup
2.Provide quota limit
3.Mount share on windows and check properties
4.Try to write data any data to it

Actual results:
Disc size is full and does not allow any data

Expected results:
Should show proper info and should not block IO

Additional info:

Comment 4 Vivek Das 2016-11-29 06:26:12 UTC

Moving on, i disable and enable quota after removing all quota limitation.
Now when i set quota limitation again on root share & subfolder. The "gluster volume quota VOLNAME list" command showed no data i.e NA in all the sections (soft limit, hard limit, used etc) of the list.

After using the setup for almost 15minutes that is creating files in the share and doing basic operation the "gluster volume quota VOLNAME list" command started showing correct data and is working fine for now.

So somewhere the quota accounting is getting messed up. Sanoj can you please look in to this and provide information regarding how did it showed 16384 PB of used space in first place.

Comment 5 Sanoj Unnikrishnan 2016-11-30 05:24:35 UTC

As we saw that initial bits of trusted.glusterfs.quota.size xattr on the bricks were FFFFFFFFFFF, Its evident that the accounting is wrong. The remaining bits representing file count/dir count were correct.
We need to know if the issue is with quota crawler (which populates xattrs initially) or the xattr update or was there a race between these two? 
Since Quota disable followed by enable resolved the issue - looks like crawler alone may not have led  to the issue.
To RCA further, we need to know,
1) Was there any io in progess while quota was being enabled ?
2) Was there any operations done between step 1 and 2 ?

Comment 8 Vivek Das 2016-12-01 08:29:08 UTC

(In reply to Sanoj Unnikrishnan from comment #5)
> As we saw that initial bits of trusted.glusterfs.quota.size xattr on the
> bricks were FFFFFFFFFFF, Its evident that the accounting is wrong. The
> remaining bits representing file count/dir count were correct.
> We need to know if the issue is with quota crawler (which populates xattrs
> initially) or the xattr update or was there a race between these two? 
> Since Quota disable followed by enable resolved the issue - looks like
> crawler alone may not have led  to the issue.
> To RCA further, we need to know,
> 1) Was there any io in progess while quota was being enabled ?

    No there was no IO in progress.

> 2) Was there any operations done between step 1 and 2 ?

    No operations were performed between step1 & step2. There was just 1 empty directory present in the share way before enabling quota.

Comment 12 Sanoj Unnikrishnan 2017-09-22 05:07:44 UTC

We don't have a clear RCA or reproducer yet for this.