Bug 1413094

Summary: [GSS] df -h not matching with quota space available
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Cal Calhoun <ccalhoun>
Component: quotaAssignee: Sanoj Unnikrishnan <sunnikri>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Rahul Hinduja <rhinduja>
Severity: high Docs Contact:
Priority: low    
Version: rhgs-3.1CC: ccalhoun, nbalacha, pousley, rcyriac, rgowdapp, rhs-bugs, sasundar, storage-qa-internal, sunnikri, vbellur
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: Accounting
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-15 03:04:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
xattr_analysis
none
log accounting script none

Description Cal Calhoun 2017-01-13 15:35:05 UTC
Description of problem:

Output from du -sh does not agree with the reported free space from gluster quota list.

[root@node1 ~]# gluster volume quota volume1 list
Path    Hard-limit  Soft-limit   Used    Available  Soft-limit exceeded? Hard-limit exceeded?
----------------------------------------------------------------------------------------------------
/node1  5.0TB       80%(4.0TB)   3.5TB   1.5TB      No                          No

Actual usage from hosts perspective:-
/mount
[root@host1 mount]# du -sh .
593G    .

No split-brain issues
[root@node2 ~]# gluster volume heal volume1 info split-brain
Brick brick1-nfs:/rhs/xfs/brick1
Status: Connected
Number of entries in split-brain: 0

Brick brick2-nfs:/rhs/xfs/brick1
Status: Connected
Number of entries in split-brain: 0

Version-Release number of selected component (if applicable):

  glusterfs-3.7.9-10.el6rhs.x86_64

Comment 2 SATHEESARAN 2017-01-18 06:02:06 UTC
Moving the component to 'quota' as this issue seems to be with quota

Comment 3 Sanoj Unnikrishnan 2017-01-24 11:33:07 UTC
Hi cal, 

There are 2 metrics here, Used space and available space.
a) du -sh will give the used space (not effected by quota limit).
this should approx match with the used space given by quota list on /

There is a discrepancy here i believe 593G vs 3.5T

b) df -h should give the available space. When quota limit is enabled (with deem-statfs option on) then the available space should honor the quota limit.
The available space on df -h does seems to match with limit-set (1.5 T) 

without deem-statfs it shows 19T

#df -h 
Filesystem Size Used Avail Use% Mounted on
 cuarhstor-vip:/oracle_backups/cuaorc001a 5.0T 3.6T 1.5T 71% /backup 

# gluster volume set oracle_backups features.quota-deem-statfs off
Back on host 
cuarhstor-vip:/oracle_backups/cuaorc001a 25T 5.9T 19T 24% /backup

So, I would like you to confirm if used space accounting(a) is indeed the problem.
If that is the case then, it would help to know :
1) backend xattr value on the bricks.
  # getfattr  -d -m. -e hex path 
  we need this detail for path with mismatch of used space

2) what operations were being run that led to the scenario 


From the logs I also see , 
[2016-05-29 07:14:42.213879] W [quota.c:3439:quota_statfs_validate_cbk] 0-oracle_backups-quota: quota context is not present in inode (gfid:00000000-0000-0000-0000-000000000001)
[2016-05-29 07:14:45.322440] W [marker-quota.c:2070:mq_initiate_quota_txn] (-->/usr/lib64/glusterfs/3.4.0.44rhs/xlator/features/locks.so(pl_truncate_cbk+0xf1) [0x7f399ecb25f1] (-->/usr/lib64/glusterfs/3.4.0.44rhs/xlator/performance/io-threads.so(iot_ftruncate_cbk+0xcc) [0x7f399ea93c9c] (-->/usr/lib64/glusterfs/3.4.0.44rhs/xlator/features/marker.so(marker_ftruncate_cbk+0x16c) [0x7f399e66f3dc]))) 0-oracle_backups-marker: could not allocate  contribution node for (<gfid:b5d0b684-5f33-4efd-a281-dfa784f30349>) parent: ((null))
[2016-05-29 12:01:05.125254] W [quota.c:3447:quota_statfs_validate_cbk] 0-oracle_backups-quota: size key not present in dict

Need to see why quota context is not available for root, 
When size key is not avaialable, there seems to be a bug in quota_statfs_validate_cbk , where we incorrectly update the quota context with wrong size. I will look further into this.

Comment 4 Cal Calhoun 2017-01-25 17:11:02 UTC
Hello Sanoj;

  Thanks for looking into this.  What exactly do you need from me to 'confirm if used space accounting is indeed the problem.'?  Are you just interested in the problem from their perspective?  Whether it's the used space or free space they're concerned with, or is there a specific test that you would like them to perform to determine the answer to your question?

Comment 5 Sanoj Unnikrishnan 2017-01-25 18:21:33 UTC
(In reply to Cal Calhoun from comment #4)
> Hello Sanoj;
> 
>   Thanks for looking into this.  What exactly do you need from me to
> 'confirm if used space accounting is indeed the problem.'?  Are you just
> interested in the problem from their perspective?  Whether it's the used
> space or free space they're concerned with, or is there a specific test that
> you would like them to perform to determine the answer to your question?

Correct, both actually. Wanted to know user perspective as well as the output of getfattr command on dirs where quota list accounting does not match with du accounting. 

getfattr  -d -m. -e hex <brick_path>/node1

Comment 8 Cal Calhoun 2017-03-17 18:37:31 UTC
@ Sanoj: Yes, the customer indicates that the output of the du -sh command run against the brick reflects approximately what they believe the correct size should be.

Thanks for looking into this.

Comment 17 Sanoj Unnikrishnan 2017-12-11 08:59:50 UTC
Created attachment 1365825 [details]
xattr_analysis

Attaching scripts to determine if the issue is with accounting

Comment 18 Sanoj Unnikrishnan 2017-12-11 09:00:19 UTC
Created attachment 1365826 [details]
log accounting script