Description of problem: ======================= Once the quota limit was set to 1GB, created 2000 files each of 1MB (Total size=2GB). File creation were successful up-to 1500 files (1.5GB) size and the rest complained that disk quota exceeded. Removed all the entries from the client but "quota list" command says 13M used. Ideally it should show 0M used. Here is the output: =================== [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 13.0MB 1011.0MB [root@dj ~]# Version-Release number of selected component (if applicable): ============================================================= glusterfs-server-3.4.0.35rhs-1.el6rhs.x86_64 How reproducible: ================= Intermittent, could reproduce in 2 tries Steps to Reproduce: =================== 1. Created a new 6*2 volume (vol-dr) 2. Enabled the quota 3. Set the quota limit to (1GB) 4. Mounted on client (NFS) 5. Created 2000 files each of 1M from the mount point using for i in {1..2000} ; do dd if=/dev/input_file of=file.$i bs=1M count=1 ; done Note: /dev/input_file is created using dd if=/dev/urandom of=/dev/input_file bs=1M count=1024 6. Approx 1500 file creation were successful and than it complains Disk Quota Exceeded 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166908 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184721 s, 56.8 MB/s dd: opening `file.1552': Disk quota exceeded 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250222 s, 41.9 MB/s dd: opening `file.1554': Disk quota exceeded 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0575855 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0278467 s, 37.7 MB/s dd: opening `file.1557': Disk quota exceeded dd: opening `file.1558': Disk quota exceeded dd: opening `file.1559': Disk quota exceeded dd: opening `file.1560': Disk quota exceeded dd: opening `file.1561': Disk quota exceeded dd: opening `file.1562': Disk quota exceeded 7. Checked the size used as [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 1.5GB 0Bytes [root@dj ~]# 8. Removed all the files from mount point using "rm -rf *" [root@tia nfs]# rm -rf * [root@tia nfs]# ls -l total 0 [root@tia nfs]# [root@tia nfs]# du -sh 0 . [root@tia nfs]# 9. All files are removed from client, but the quota list shows 13M used: [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 13.0MB 1011.0MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 13.0MB 1011.0MB [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 13.0MB 1011.0MB [root@dj ~]# Expected results: ================= [root@dj ~]# gluster volume quota vol-dr list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 0MB 1.0GB [root@dj ~]# Additional info: ================ Result is same even after 15-20 mins
[root@dj ~]# getfattr -d -e hex -m . /rhs/brick1/b* getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/b1 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000555555547ffffffd trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000000000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 # file: rhs/brick1/b3 trusted.afr.vol-dr-client-2=0x000000000000000000000000 trusted.afr.vol-dr-client-3=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000400000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 # file: rhs/brick1/b5 trusted.afr.vol-dr-client-4=0x000000000000000000000000 trusted.afr.vol-dr-client-5=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000400000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 [root@dj ~]# [root@fan ~]# getfattr -d -e hex -m . /rhs/brick1/b* getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/b2 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000555555547ffffffd trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000000000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 # file: rhs/brick1/b4 trusted.afr.vol-dr-client-2=0x000000000000000000000000 trusted.afr.vol-dr-client-3=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000400000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 # file: rhs/brick1/b6 trusted.afr.vol-dr-client-4=0x000000000000000000000000 trusted.afr.vol-dr-client-5=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000400000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 [root@fan ~]# [root@mia ~]# getfattr -d -e hex -m . /rhs/brick1/b* getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/b11 trusted.afr.vol-dr-client-10=0x000000000000000000000000 trusted.afr.vol-dr-client-11=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000000000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 # file: rhs/brick1/b7 trusted.afr.vol-dr-client-6=0x000000000000000000000000 trusted.afr.vol-dr-client-7=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000400000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 # file: rhs/brick1/b9 trusted.afr.vol-dr-client-8=0x000000000000000000000000 trusted.afr.vol-dr-client-9=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000100000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 [root@mia ~]# [root@wingo ~]# getfattr -d -e hex -m . /rhs/brick1/b* getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/b10 trusted.afr.vol-dr-client-8=0x000000000000000000000000 trusted.afr.vol-dr-client-9=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000100000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 # file: rhs/brick1/b12 trusted.afr.vol-dr-client-10=0x000000000000000000000000 trusted.afr.vol-dr-client-11=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000000000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 # file: rhs/brick1/b8 trusted.afr.vol-dr-client-6=0x000000000000000000000000 trusted.afr.vol-dr-client-7=0x000000000000000000000000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000400000 trusted.glusterfs.volume-id=0xb6fb5907764b44d482b8302a7e1c3be6 [root@wingo ~]#
I don't see this with glusterfs-server-3.4.0.40rhs-1.el6rhs.x86_64 [shanks@localhost Music]$ df -h . Filesystem Size Used Avail Use% Mounted on 10.70.43.3:/shanks-quota 1.0G 1.0G 0 100% /home [shanks@localhost Music]$ rm -fr * [shanks@localhost Music]$ df -h . Filesystem Size Used Avail Use% Mounted on 10.70.43.3:/shanks-quota 1.0G 0 1.0G 0% /home [shanks@localhost Music]$ [root@server1 ~]# gluster vol quota shanks-quota list /shanks/Music Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /shanks/Music 1.0GB 80% 0Bytes 1.0GB [root@server1 ~]#
based on comment#4, moving this BZ to verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html