1025333 – quota: after reset command, and setting quota-deem-statfs to "on", df displays wrong values

Bug 1025333 - quota: after reset command, and setting quota-deem-statfs to "on", df displays wrong values

Summary: quota: after reset command, and setting quota-deem-statfs to "on", df display...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat
Component:	glusterd
Sub Component:
Version:	2.1
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	urgent
Target Milestone:	---
Target Release:	---
Assignee:	Bug Updates Notification Mailing List
QA Contact:	Saurabh
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2013-10-31 13:55 UTC by Saurabh
Modified:	2016-01-19 06:13 UTC (History)
CC List:	5 users (show)
Fixed In Version:	glusterfs-3.4.0.38rhs
Doc Type:	Bug Fix
Doc Text:	Previously, when data is present in the volume before the quota feature is enabled and quota limits are set on it, such that the data already present consumes disk space greater than the limit, df utility would report negative values for "Used" and "Available" columns. Now, with this update, df utility reports properly with data populated before quota feature is enabled.
Clone Of:
Environment:
Last Closed:	2013-11-27 15:45:06 UTC
Target Upstream Version:
Dependent Products:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2013:1769	0	normal	SHIPPED_LIVE	Red Hat Storage 2.1 enhancement and bug fix update #1	2013-11-27 20:17:39 UTC

Description Saurabh 2013-10-31 13:55:40 UTC

Description of problem:
The issue happens is that df command shows wrong values.
"gluster volume reset $volname" command is used for setting the volume level 
options to their respective default values.

The volume level options also include quota-deem-statfs. quota-deem-statfs enables standard tools like "df" to show quota stats for a path.

When we use "gluster volume reset command", quota-deem-statfs set the value of this option to default to off. Now, in order to use standard tools like "df" one needs to set quota-deem-statfs back to "on", i.e. where the problem is seen.

As, the df command displays wrong values or, rather it itseld becomes confused


Version-Release number of selected component (if applicable):
glusterfs-3.4.0.37rhs

How reproducible:
happen to be seen on this build, as reset seems to be working from this build

Steps to Reproduce:
1. create a volume, start it
2. enable quota, put quota-deem-statfs to on.
3. set some limit on the root of the volume. say "10GB"
4. mount over nfs, create a dir.
5. set a limit on dir, say 5GB
6. create data inside the dir.
7. gluster volume quota $volname list,
[root@quota1 ~]# gluster volume quota dist-rep5 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir                                       5.0GB       80%       5.0GB  0Bytes
/                                         10.0GB       80%       5.0GB   5.0GB

8. gluster volume info
 [root@quota1 ~]# gluster volume info dist-rep5
 
Volume Name: dist-rep5
Type: Distributed-Replicate
Volume ID: 48279380-8783-4cc6-812b-916db4a56fe7
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.42.186:/rhs/brick1/d1r15
Brick2: 10.70.43.181:/rhs/brick1/d1r25
Brick3: 10.70.43.18:/rhs/brick1/d2r15
Brick4: 10.70.43.22:/rhs/brick1/d2r25
Brick5: 10.70.42.186:/rhs/brick1/d3r15
Brick6: 10.70.43.181:/rhs/brick1/d3r25
Brick7: 10.70.43.18:/rhs/brick1/d4r15
Brick8: 10.70.43.22:/rhs/brick1/d4r25
Brick9: 10.70.42.186:/rhs/brick1/d5r15
Brick10: 10.70.43.181:/rhs/brick1/d5r25
Brick11: 10.70.43.18:/rhs/brick1/d6r15
Brick12: 10.70.43.22:/rhs/brick1/d6r25
Options Reconfigured:
features.quota-deem-statfs: on
diagnostics.client-log-level: DEBUG
features.quota: on

9. gluster volume reset $volname
[root@quota3 ~]# gluster volume reset dist-rep5
volume reset: success: All unprotected fields were reset. To reset the protected fields, use 'force'.

10. gluster volume info
[root@quota3 ~]# gluster volume info dist-rep5
 
Volume Name: dist-rep5
Type: Distributed-Replicate
Volume ID: 48279380-8783-4cc6-812b-916db4a56fe7
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.42.186:/rhs/brick1/d1r15
Brick2: 10.70.43.181:/rhs/brick1/d1r25
Brick3: 10.70.43.18:/rhs/brick1/d2r15
Brick4: 10.70.43.22:/rhs/brick1/d2r25
Brick5: 10.70.42.186:/rhs/brick1/d3r15
Brick6: 10.70.43.181:/rhs/brick1/d3r25
Brick7: 10.70.43.18:/rhs/brick1/d4r15
Brick8: 10.70.43.22:/rhs/brick1/d4r25
Brick9: 10.70.42.186:/rhs/brick1/d5r15
Brick10: 10.70.43.181:/rhs/brick1/d5r25
Brick11: 10.70.43.18:/rhs/brick1/d6r15
Brick12: 10.70.43.22:/rhs/brick1/d6r25
Options Reconfigured:
features.quota: on

11. create one more directory inside the mountpoint, dont set any limit on this dir, now create data inside this volume till quota limit is reached.

12.
[root@quota1 ~]# gluster volume quota dist-rep5 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir                                       5.0GB       80%       5.0GB  0Bytes
/                                         10.0GB       80%      10.1GB  0Bytes


13. gluster volume set $volname quota-deem-statfs on

14. df -h
[root@quota1 ~]# df -h /var/run/gluster/dist-rep5
Filesystem            Size  Used Avail Use% Mounted on
localhost:dist-rep5    10G  -16E   16E   -  /var/run/gluster/dist-rep5




Actual results:
result as seen in step 14, 

result is same on client and server both.

Expected results:
after reset, putting quota-deem-statfs to "on" should let df to display the quota stats correctly.

Additional info:

Comment 1 Saurabh 2013-10-31 14:04:34 UTC

for cluster having nodes namely, quota[1-4]

from quota1,
[root@quota1 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r15
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/d1r15
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000125841000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

# file: rhs/brick1/d3r15
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

# file: rhs/brick1/d5r15
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

for directory,
[root@quota1 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r15/dir
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/d1r15/dir
trusted.afr.dist-rep5-client-0=0x000000000000000000000000
trusted.afr.dist-rep5-client-1=0x000000000000000000000000
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000000a3f31000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x00000000a3f31000

# file: rhs/brick1/d3r15/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000

# file: rhs/brick1/d5r15/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000


from quota2,
[root@quota2 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r25
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/d1r25
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000125920000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

# file: rhs/brick1/d3r25
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

# file: rhs/brick1/d5r25
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

for directory,
[root@quota2 ~]# 
[root@quota2 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r25/dir
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/d1r25/dir
trusted.afr.dist-rep5-client-0=0x000000000000000000000000
trusted.afr.dist-rep5-client-1=0x000000000000000000000000
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000000a4010000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x00000000a4010000

# file: rhs/brick1/d3r25/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000

# file: rhs/brick1/d5r25/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000


for quota3,
[root@quota3 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r15
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/d2r15
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x000000015e000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

# file: rhs/brick1/d4r15
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

# file: rhs/brick1/d6r15
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

[root@quota3 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r15/dir
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/d2r15/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x000000009c400000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x000000009c400000

# file: rhs/brick1/d4r15/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000

# file: rhs/brick1/d6r15/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000


for quota4, 
[root@quota4 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r25
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/d2r25
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x000000015e000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

# file: rhs/brick1/d4r25
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

# file: rhs/brick1/d6r25
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000280000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x4827938087834cc6812b916db4a56fe7

for directory,
[root@quota4 ~]# getfattr -m . -d -e hex /rhs/brick1/d*r25/dir
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/d2r25/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x000000009c400000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x000000009c400000

# file: rhs/brick1/d4r25/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000

# file: rhs/brick1/d6r25/dir
trusted.gfid=0x7acf1b2a6a7147f39d9690e332d750f7
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000140000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000


collect sosreports from here,
http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1025333/

Comment 3 Saurabh 2013-11-04 10:32:45 UTC

verified on glusterfs-3.4.0.38rhs

as can be seen here,

[root@quota1 ~]# df -h /var/run/gluster/dist-rep5
Filesystem            Size  Used Avail Use% Mounted on
localhost:dist-rep5    15G   15G     0 100% /var/run/gluster/dist-rep5
[root@quota1 ~]# gluster volume reset dist-rep5
volume reset: success: All unprotected fields were reset. To reset the protected fields, use 'force'.
[root@quota1 ~]# df -h /var/run/gluster/dist-rep5
Filesystem            Size  Used Avail Use% Mounted on
localhost:dist-rep5   2.2T  2.1T   63G  98% /var/run/gluster/dist-rep5
[root@quota1 ~]# gluster volume set dist-rep5 quota-deem-statfs on
volume set: success
[root@quota1 ~]# 
[root@quota1 ~]# df -h /var/run/gluster/dist-rep5
Filesystem            Size  Used Avail Use% Mounted on
localhost:dist-rep5    15G   15G     0 100% /var/run/gluster/dist-rep5

Comment 4 errata-xmlrpc 2013-11-27 15:45:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html

Note You need to log in before you can comment on or make changes to this bug.