Bug 1021492 - quota: Used space is non 0 after deletion of all files from the client
quota: Used space is non 0 after deletion of all files from the client
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.1
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
Saurabh
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-21 07:56 EDT by Rahul Hinduja
Modified: 2016-01-19 01:13 EST (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.40rhs
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-27 10:43:08 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rahul Hinduja 2013-10-21 07:56:33 EDT
Description of problem:
=======================

Once the quota limit was set to 1GB, created 2000 files each of 1MB (Total size=2GB). File creation were successful up-to 1500 files (1.5GB) size and the rest complained that disk quota exceeded. Removed all the entries from the client but "quota list" command says 13M used. Ideally it should show 0M used.

Here is the output:
===================

[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      13.0MB 1011.0MB
[root@dj ~]# 

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-server-3.4.0.35rhs-1.el6rhs.x86_64


How reproducible:
=================

Intermittent, could reproduce in 2 tries

Steps to Reproduce:
===================
1. Created a new 6*2 volume (vol-dr)
2. Enabled the quota
3. Set the quota limit to (1GB)
4. Mounted on client (NFS)
5. Created 2000 files each of 1M from the mount point using

for i in {1..2000} ; do dd if=/dev/input_file of=file.$i bs=1M count=1 ; done

Note: /dev/input_file is created using

dd if=/dev/urandom of=/dev/input_file bs=1M count=1024

6. Approx 1500 file creation were successful and than it complains Disk Quota Exceeded

1+0 records out
1048576 bytes (1.0 MB) copied, 0.0166908 s, 62.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0184721 s, 56.8 MB/s
dd: opening `file.1552': Disk quota exceeded
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0250222 s, 41.9 MB/s
dd: opening `file.1554': Disk quota exceeded
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0575855 s, 18.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0278467 s, 37.7 MB/s
dd: opening `file.1557': Disk quota exceeded
dd: opening `file.1558': Disk quota exceeded
dd: opening `file.1559': Disk quota exceeded
dd: opening `file.1560': Disk quota exceeded
dd: opening `file.1561': Disk quota exceeded
dd: opening `file.1562': Disk quota exceeded

7. Checked the size used as

[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%       1.5GB  0Bytes
[root@dj ~]# 

8. Removed all the files from mount point using "rm -rf *"

[root@tia nfs]# rm -rf *
[root@tia nfs]# ls -l
total 0
[root@tia nfs]# 
[root@tia nfs]# du -sh 
0	.
[root@tia nfs]# 

9. All files are removed from client, but the quota list shows 13M used:

[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      13.0MB 1011.0MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      13.0MB 1011.0MB
[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      13.0MB 1011.0MB
[root@dj ~]# 




Expected results:
=================

[root@dj ~]# gluster volume quota vol-dr list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%      0MB    1.0GB
[root@dj ~]# 

Additional info:
================

Result is same even after 15-20 mins
Comment 2 Rahul Hinduja 2013-10-21 08:08:37 EDT
[root@dj ~]# getfattr -d -e hex -m . /rhs/brick1/b*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/b1
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

# file: rhs/brick1/b3
trusted.afr.vol-dr-client-2=0x000000000000000000000000
trusted.afr.vol-dr-client-3=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000400000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

# file: rhs/brick1/b5
trusted.afr.vol-dr-client-4=0x000000000000000000000000
trusted.afr.vol-dr-client-5=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000400000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

[root@dj ~]# 


[root@fan ~]# getfattr -d -e hex -m . /rhs/brick1/b*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/b2
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

# file: rhs/brick1/b4
trusted.afr.vol-dr-client-2=0x000000000000000000000000
trusted.afr.vol-dr-client-3=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000400000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

# file: rhs/brick1/b6
trusted.afr.vol-dr-client-4=0x000000000000000000000000
trusted.afr.vol-dr-client-5=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000400000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

[root@fan ~]# 


[root@mia ~]# getfattr -d -e hex -m . /rhs/brick1/b*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/b11
trusted.afr.vol-dr-client-10=0x000000000000000000000000
trusted.afr.vol-dr-client-11=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

# file: rhs/brick1/b7
trusted.afr.vol-dr-client-6=0x000000000000000000000000
trusted.afr.vol-dr-client-7=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000400000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

# file: rhs/brick1/b9
trusted.afr.vol-dr-client-8=0x000000000000000000000000
trusted.afr.vol-dr-client-9=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000100000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

[root@mia ~]# 


[root@wingo ~]# getfattr -d -e hex -m . /rhs/brick1/b*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/b10
trusted.afr.vol-dr-client-8=0x000000000000000000000000
trusted.afr.vol-dr-client-9=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000100000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

# file: rhs/brick1/b12
trusted.afr.vol-dr-client-10=0x000000000000000000000000
trusted.afr.vol-dr-client-11=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

# file: rhs/brick1/b8
trusted.afr.vol-dr-client-6=0x000000000000000000000000
trusted.afr.vol-dr-client-7=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000400000
trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6

[root@wingo ~]#
Comment 4 Gowrishankar Rajaiyan 2013-11-09 05:36:54 EST
I don't see this with glusterfs-server-3.4.0.40rhs-1.el6rhs.x86_64

[shanks@localhost Music]$ df -h .
Filesystem                Size  Used Avail Use% Mounted on
10.70.43.3:/shanks-quota  1.0G  1.0G     0 100% /home
[shanks@localhost Music]$ rm -fr *
[shanks@localhost Music]$ df -h .
Filesystem                Size  Used Avail Use% Mounted on
10.70.43.3:/shanks-quota  1.0G     0  1.0G   0% /home
[shanks@localhost Music]$ 


[root@server1 ~]# gluster vol quota shanks-quota list /shanks/Music
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/shanks/Music                              1.0GB       80%      0Bytes   1.0GB
[root@server1 ~]#
Comment 5 Saurabh 2013-11-11 04:06:09 EST
based on comment#4, moving this BZ to verified
Comment 6 errata-xmlrpc 2013-11-27 10:43:08 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html

Note You need to log in before you can comment on or make changes to this bug.