1002885 – Quota limit violated upon creating a file after add-brick and fix-layout

Bug 1002885 - Quota limit violated upon creating a file after add-brick and fix-layout

Summary: Quota limit violated upon creating a file after add-brick and fix-layout

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	glusterfs
Sub Component:
Version:	2.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	vpshastry
QA Contact:	shylesh
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1021949 (view as bug list)
Depends On:
Blocks:	957769 1003549
TreeView+	depends on / blocked

Reported:	2013-08-30 07:22 UTC by shylesh
Modified:	2014-08-11 23:22 UTC (History)
CC List:	10 users (show)
Fixed In Version:	glusterfs-3.4.0.38rhs
Doc Type:	Bug Fix
Doc Text:	Previously, the quota limit configuration stored in the extended attributes of the directory would not be present in the newly added brick. The quota list command may fail to list the limits on the directory when only the newly added bricks are available. Now, with this update, the configuration is copied on the newly added bricks and the quota list command does not fail.
Clone Of:
Environment:
Last Closed:	2013-11-27 15:35:56 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2013:1769	0	normal	SHIPPED_LIVE	Red Hat Storage 2.1 enhancement and bug fix update #1	2013-11-27 20:17:39 UTC

Description shylesh 2013-08-30 07:22:09 UTC

Description of problem:
After doing add-brick and fix-layout on a distributed-replicate volume which has already reached the quota limit, file which gets hashed to the newly added bricks will violate quota

Version-Release number of selected component (if applicable):
[root@gqac023 mnt]# rpm -qa| grep gluster
glusterfs-fuse-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-server-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-api-devel-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-libs-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-api-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-devel-3.4.0.20rhsquota5-1.el6rhs.x86_64


How reproducible:
always

Steps to Reproduce:
1.created a distributed-replica volume of 3x2 configuration and set the hard quota limit to 1GB on /

2.Mounted the volume and created a directory and set the quota to 1GB and inside this directory created a file of size 1GB

3.now further file creation fails with "quota exceeded" message

4. added one more pair of brick and ran "rebalance fix-layout"

5. Now tried to create more files
 
Actual results:

The files which gets hashed to the newly added pair will not respect quota as they get crated and violating quota
Expected results:


Additional info:

Volume Name: replica
Type: Distributed-Replicate
Volume ID: fba43d74-bbd0-407e-beda-67ebeb4fb9c9
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: gqac022.sbu.lab.eng.bos.redhat.com:/home/r0
Brick2: gqac023.sbu.lab.eng.bos.redhat.com:/home/r0
Brick3: gqac024.sbu.lab.eng.bos.redhat.com:/home/r1
Brick4: gqac025.sbu.lab.eng.bos.redhat.com:/home/r1
Brick5: gqac022.sbu.lab.eng.bos.redhat.com:/home/r2
Brick6: gqac023.sbu.lab.eng.bos.redhat.com:/home/r2
Brick7: gqac024.sbu.lab.eng.bos.redhat.com:/home/r3 ===>newly added pair
Brick8: gqac025.sbu.lab.eng.bos.redhat.com:/home/r3 ===>newly added pair
Options Reconfigured:
features.quota: on
[root@gqac023 mnt]# gluster v quota replica  list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB 8977070246986897596       3.8GB  0Bytes
/test                                      1.0GB 8977070246986897596       3.8GB  0Bytes


Cluster info
===========
RHS nodes
---------
gqac022.sbu.lab.eng.bos.redhat.com
gqac023.sbu.lab.eng.bos.redhat.com
gqac024.sbu.lab.eng.bos.redhat.com
gqac025.sbu.lab.eng.bos.redhat.com


Mounted on 
==========
gqac022.sbu.lab.eng.bos.redhat.com

Mount point
============
/mnt1



xattrs from the bricks for /
============================


pair1
=====
[root@gqac022 test]# getfattr -d -m . -e hex /home/r0/
getfattr: Removing leading '/' from absolute path names
# file: home/r0/
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000003ffffffe
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x00000000046f0000
trusted.glusterfs.volume-id=0xfba43d74bbd0407ebeda67ebeb4fb9c9

[root@gqac023 mnt]# getfattr -d -m . -e hex /home/r0
getfattr: Removing leading '/' from absolute path names
# file: home/r0
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000003ffffffe
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x00000000046f0000
trusted.glusterfs.volume-id=0xfba43d74bbd0407ebeda67ebeb4fb9c9


pair 2
======
[root@gqac024 in]# getfattr -d -m . -e hex /home/r1
getfattr: Removing leading '/' from absolute path names
# file: home/r1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x000000003b9aca00
trusted.glusterfs.volume-id=0xfba43d74bbd0407ebeda67ebeb4fb9c9

[root@gqac025 home]# getfattr -d -m . -e hex /home/r1
getfattr: Removing leading '/' from absolute path names
# file: home/r1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x000000003b9aca00
trusted.glusterfs.volume-id=0xfba43d74bbd0407ebeda67ebeb4fb9c9

pair3
=====
[root@gqac022 test]# getfattr -d -m . -e hex /home/r2
getfattr: Removing leading '/' from absolute path names
# file: home/r2
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.volume-id=0xfba43d74bbd0407ebeda67ebeb4fb9c9

[root@gqac023 mnt]# getfattr -d -m . -e hex /home/r2
getfattr: Removing leading '/' from absolute path names
# file: home/r2
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.volume-id=0xfba43d74bbd0407ebeda67ebeb4fb9c9


pair 4
======
[root@gqac024 in]# getfattr -d -m . -e hex /home/r3
getfattr: Removing leading '/' from absolute path names
# file: home/r3
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x00000000b2d05e00
trusted.glusterfs.volume-id=0xfba43d74bbd0407ebeda67ebeb4fb9c9

[root@gqac025 home]# getfattr -d -m . -e hex /home/r3
getfattr: Removing leading '/' from absolute path names
# file: home/r3
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x00000000b2d05e00
trusted.glusterfs.volume-id=0xfba43d74bbd0407ebeda67ebeb4fb9c9




xattrs dump for /test
=====================
pair1
=====
[root@gqac022 test]# getfattr -d -m . -e hex /home/r0/test
getfattr: Removing leading '/' from absolute path names
# file: home/r0/test
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x86f12f86ab6a4700be9d894faf33a450
trusted.glusterfs.dht=0x0000000100000000000000003ffffffe
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000000046f0000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x00000000046f0000

[root@gqac023 mnt]# getfattr -d -m . -e hex /home/r0/test
getfattr: Removing leading '/' from absolute path names
# file: home/r0/test
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x86f12f86ab6a4700be9d894faf33a450
trusted.glusterfs.dht=0x0000000100000000000000003ffffffe
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000000046f0000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x00000000046f0000


pair2
=====
[root@gqac024 in]# getfattr -d -m . -e hex /home/r1/test
getfattr: Removing leading '/' from absolute path names
# file: home/r1/test
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x86f12f86ab6a4700be9d894faf33a450
trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x000000003b9aca00
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x000000003b9aca00

[root@gqac025 home]# getfattr -d -m . -e hex /home/r1/test
getfattr: Removing leading '/' from absolute path names
# file: home/r1/test
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x86f12f86ab6a4700be9d894faf33a450
trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x000000003b9aca00
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x000000003b9aca00

pair3
=======
[root@gqac022 test]# getfattr -d -m . -e hex /home/r2/test
getfattr: Removing leading '/' from absolute path names
# file: home/r2/test
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x86f12f86ab6a4700be9d894faf33a450
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x0000000000000000

[root@gqac023 mnt]# getfattr -d -m . -e hex /home/r2/test
getfattr: Removing leading '/' from absolute path names
# file: home/r2/test
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x86f12f86ab6a4700be9d894faf33a450
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x00000000400000007c94f5d3bb61d8bc
trusted.glusterfs.quota.size=0x0000000000000000


pair4
=======
[root@gqac024 in]# getfattr -d -m . -e hex /home/r3/test
getfattr: Removing leading '/' from absolute path names
# file: home/r3/test
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x86f12f86ab6a4700be9d894faf33a450
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000000b2d05e00
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x00000000b2d05e00


[root@gqac025 home]# getfattr -d -m . -e hex /home/r3/test
getfattr: Removing leading '/' from absolute path names
# file: home/r3/test
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x86f12f86ab6a4700be9d894faf33a450
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000000b2d05e00
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x00000000b2d05e00



========================================================================

the files which are hashed to pair4 are violatoing quota


[root@gqac022 test]# dd if=/dev/urandom of=file23 bs=1GB count=1
dd: opening `file23': Disk quota exceeded
[root@gqac022 test]# dd if=/dev/urandom of=file24 bs=1GB count=1
dd: opening `file24': Disk quota exceeded
[root@gqac022 test]# dd if=/dev/urandom of=file25 bs=1GB count=1
dd: opening `file25': Disk quota exceeded
[root@gqac022 test]# dd if=/dev/urandom of=file26 bs=1GB count=1
1+0 records in
1+0 records out
1000000000 bytes (1.0 GB) copied, 130.736 s, 7.6 MB/s
You have new mail in /var/spool/mail/root
[root@gqac022 test]# dd if=/dev/urandom of=file27 bs=1GB count=1
1+0 records in
1+0 records out
1000000000 bytes (1.0 GB) copied, 130.923 s, 7.6 MB/s
[root@gqac022 test]# dd if=/dev/urandom of=file28 bs=1GB count=1
dd: opening `file28': Disk quota exceeded
You have new mail in /var/spool/mail/root
[root@gqac022 test]# dd if=/dev/urandom of=file29 bs=1GB count=1
1+0 records in
1+0 records out
1000000000 bytes (1.0 GB) copied, 131.003 s, 7.6 MB/s






attached the sosreports

Comment 3 Raghavendra Bhat 2013-09-17 10:50:21 UTC

(https://code.engineering.redhat.com/gerrit/#/c/12366/) and https://code.engineering.redhat.com/gerrit/#/c/12397/ have been submitted for review.

Comment 4 Gowrishankar Rajaiyan 2013-10-17 09:55:52 UTC

Per bug triage 10/17.

Comment 6 senaik 2013-10-23 06:21:57 UTC

*** Bug 1021949 has been marked as a duplicate of this bug. ***

Comment 7 shylesh 2013-11-13 05:15:37 UTC

verified on 3.4.0.43rhs-1.el6rhs.x86_64.

Now adding a brick properly heals quota xattrs thus quota is preserved.

Comment 9 errata-xmlrpc 2013-11-27 15:35:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html

Note You need to log in before you can comment on or make changes to this bug.