1341190 – conservative merge happening on a x3 volume for a deleted file

Bug 1341190 - conservative merge happening on a x3 volume for a deleted file

Summary: conservative merge happening on a x3 volume for a deleted file

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	replicate
Sub Component:
Version:	rhgs-3.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	urgent
Target Milestone:	---
Target Release:	RHGS 3.4.0
Assignee:	Ravishankar N
QA Contact:	Vijay Avuthu
Docs Contact:
URL:
Whiteboard:	rebase
Depends On:
Blocks:	1341650 1344559 1344561 1503134
TreeView+	depends on / blocked

Reported:	2016-05-31 12:38 UTC by Nag Pavan Chilakam
Modified:	2018-09-04 06:31 UTC (History)
CC List:	3 users (show)
Fixed In Version:	glusterfs-3.12.2-1
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1341650 (view as bug list)
Environment:
Last Closed:	2018-09-04 06:29:40 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2018:2607	0	None	None	None	2018-09-04 06:31:22 UTC

Description Nag Pavan Chilakam 2016-05-31 12:38:07 UTC

Description of problem:
=========================
When I set the object limit of a directory and bring down a brick and then exceed quota object limits and then delete an existing file, I see that a conservative merge happens for that file once the brick is brought online, instead of deleting the file from the offline brick


Version-Release number of selected component (if applicable):
=====
3.7.9-6



Steps to Reproduce:
1.created a 2x3 volume
2.enabled quota, set the hard and soft timeouts to 0
3.fuse mounted the vol
4. created a dir /dir11
5. set the quota obj-limit to 10
6. created file f{1..5}--> f1 got created on subvol-1 and remaining on sbvol2
7. brought down brick6( the 3rd brick of 2nd subvol
8. created f{6..9} using touch -->passed
9. 10th file create failed as below
touch f10
touch: cannot touch ‘f10’: Disk quota exceeded
10. now did an rm of f5
[root@dhcp35-103 dir11]# rm f5
rm: remove regular empty file ‘f5’? y
==> that means now the quota level has not exceeded limit
11. more file creates   
[root@dhcp35-103 dir11]# touch f10
[root@dhcp35-103 dir11]# touch f11
touch: cannot touch ‘f11’: Disk quota exceeded

12. bring back down brick online using start force
[root@dhcp35-103 dir11]#  ####now going to bring bricks back###

12. checked that instead of f5 getting deleted from brick6, a conserveative merge happend and f5 is created on all bricks.

==>also the quota limit has now crossed the limit from backend (heal) operation

                  Path                   Hard-limit   Soft-limit      Files       Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------

/dir11                                          10       80%(8)         10         1           0             Yes                  Yes






Getfattr details of subvol 2 :

brick4(1st of subvol2):
==>before bringing brick6 down
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

==>after bringing brick6 down and creating some files and quota limit exceeded
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

=-==>after delete of file f5
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.afr.distrepx3-client-5=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001

[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or directory


=> after created f10(which could accomdate due to deelte of f5 whih passed while f11 failed due to quota exceeded##
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000002
trusted.afr.distrepx3-client-5=0x000000000000000000000002
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

===>after bringing brick6 online using force start
[root@nchilaka-node1 glusterfs]# ####now going to bring bricks back###
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# ##brick is back online
[root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.bit-rot.version=0x0200000000000000574d56f500040650
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001




=======>getfattr for brick5(2nd brick of subvol2)
[root@nchilaka-node2 glusterfs]# ################ dir11 start##############
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
[root@nchilaka-node2 glusterfs]# #getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node2 glusterfs]# #########bringing down brick
[root@nchilaka-node2 glusterfs]# ps -ef|grep 4308
root      4197 27604  0 17:22 pts/0    00:00:00 grep 4308
[root@nchilaka-node2 glusterfs]# kill -9 4308
-bash: kill: (4308) - No such process
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ##deleted f5 and created some files and quota limit exceeded#
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# #getfattr of deleted file #######;getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root@nchilaka-node2 glusterfs]# ##sorry tried to delete f7 which was not even part of filsystem during that time
[root@nchilaka-node2 glusterfs]# ##retrying delete of f5
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ##deleted f5
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.afr.distrepx3-client-5=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001

[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or directory
[root@nchilaka-node2 glusterfs]# ##created f10 whih passed while f11 failed due to quota exceeded##
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000002
trusted.afr.distrepx3-client-5=0x000000000000000000000002
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node2 glusterfs]# ####now going to bring bricks back###
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ##brick is back online
[root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.bit-rot.version=0x0200000000000000574d56f70008f91f
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001


===>brick6 (3rd brick of subvol2) The brick which is brought down as part of test


[root@nchilaka-node4 glusterfs]# ################ dir11 start##############
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
[root@nchilaka-node4 glusterfs]# #getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node4 glusterfs]# #########bringing down brick
[root@nchilaka-node4 glusterfs]# ps -ef|grep 4308
root      4308     1  0 17:18 ?        00:00:00 /usr/sbin/glusterfsd -s 10.70.42.123 --volfile-id distrepx3.10.70.42.123.rhs-brick2-distrepx3 -p /var/lib/glusterd/vols/distrepx3/run/10.70.42.123-rhs-brick2-distrepx3.pid -S /var/run/gluster/3e4dee7a2dcfb7735da718783949470e.socket --brick-name /rhs/brick2/distrepx3 -l /var/log/glusterfs/bricks/rhs-brick2-distrepx3.log --xlator-option *-posix.glusterd-uuid=51bc4ded-1ebb-4ec3-b5a3-2b5de0b025d1 --brick-port 49162 --xlator-option distrepx3-server.listen-port=49162
root      4404 27788  0 17:22 pts/0    00:00:00 grep 4308
[root@nchilaka-node4 glusterfs]# kill -9 4308
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ##deleted f5 and created some files and quota limit exceeded#
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# #getfattr of deleted file #######;getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root@nchilaka-node4 glusterfs]# ##sorry tried to delete f7 which was not even part of filsystem during that time
[root@nchilaka-node4 glusterfs]# ##retrying delete of f5
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ##deleted f5
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root@nchilaka-node4 glusterfs]# ##created f10 whih passed while f11 failed due to quota exceeded##
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node4 glusterfs]# ####now going to bring bricks back###
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ##brick is back online
[root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.distrepx3-client-3=0x000000000000000000000000
trusted.afr.distrepx3-client-4=0x000000000000000000000000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# #############creating new vol testnew
[root@nchilaka-node4 glusterfs]# rm -rf /rhs/brick1/testnew
[root@nchilaka-node4 glusterfs]# 




===>other cmd o/p
[root@nchilaka-node1 ~]# gluster v status distrepx3
Status of volume: distrepx3
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.43.22:/rhs/brick1/distrepx3     49160     0          Y       5130 
Brick 10.70.42.188:/rhs/brick1/distrepx3    49158     0          Y       31389
Brick 10.70.42.123:/rhs/brick1/distrepx3    49161     0          Y       2296 
Brick 10.70.43.22:/rhs/brick2/distrepx3     49161     0          Y       5149 
Brick 10.70.42.188:/rhs/brick2/distrepx3    49159     0          Y       31408
Brick 10.70.42.123:/rhs/brick2/distrepx3    49162     0          Y       4776 
NFS Server on localhost                     2049      0          Y       11839
Self-heal Daemon on localhost               N/A       N/A        Y       11847
Quota Daemon on localhost                   N/A       N/A        Y       11855
NFS Server on 10.70.42.123                  2049      0          Y       4796 
Self-heal Daemon on 10.70.42.123            N/A       N/A        Y       4804 
Quota Daemon on 10.70.42.123                N/A       N/A        Y       4813 
NFS Server on 10.70.42.204                  2049      0          Y       27323
Self-heal Daemon on 10.70.42.204            N/A       N/A        Y       27331
Quota Daemon on 10.70.42.204                N/A       N/A        Y       27339
NFS Server on 10.70.42.188                  2049      0          Y       4573 
Self-heal Daemon on 10.70.42.188            N/A       N/A        Y       4581 
Quota Daemon on 10.70.42.188                N/A       N/A        Y       4590 
 
Task Status of Volume distrepx3
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@nchilaka-node1 ~]# gluster v quota distrepx3 list-obj
                  Path                   Hard-limit   Soft-limit      Files       Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------
/dir1                                           35       80%(28)         90         1           0             Yes                  Yes
/dir2                                          200       80%(160)        300         1           0             Yes                  Yes
/dir3                                         1100       80%(880)       1210         1           0             Yes                  Yes
/dir4                                           50       80%(40)         40         1           9             Yes                   No
/dir5                                          100       80%(80)        199         1           0             Yes                  Yes
/dir6                                          100       80%(80)         99         1           0             Yes                  Yes
/dir7                                           10       80%(8)         10         1           0             Yes                  Yes
/dir8                                           10       80%(8)          9         1           0             Yes                  Yes
/dir10                                          10       80%(8)          9         1           0             Yes                  Yes
/dir11                                          10       80%(8)         10         1           0             Yes                  Yes
[root@nchilaka-node1 ~]# 
[root@nchilaka-node1 ~]# 
[root@nchilaka-node1 ~]# gluster v heal distrepx3 info
Brick 10.70.43.22:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.188:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.123:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.43.22:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.188:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.123:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0


 
Volume Name: distrepx3
Type: Distributed-Replicate
Volume ID: 353f80dc-aa40-468d-b6ae-8604b79e3397
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.43.22:/rhs/brick1/distrepx3
Brick2: 10.70.42.188:/rhs/brick1/distrepx3
Brick3: 10.70.42.123:/rhs/brick1/distrepx3
Brick4: 10.70.43.22:/rhs/brick2/distrepx3
Brick5: 10.70.42.188:/rhs/brick2/distrepx3
Brick6: 10.70.42.123:/rhs/brick2/distrepx3
Options Reconfigured:
features.soft-timeout: 0
features.hard-timeout: 0
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on

Comment 2 Nag Pavan Chilakam 2016-05-31 12:39:37 UTC

raised this bug as part of qa on_qa validation of  	1332199:Self Heal fails on a replica3 volume with 'disk quota exceeded'

Comment 9 Vijay Avuthu 2018-04-19 12:12:27 UTC

Update:
==========

Build Used : glusterfs-3.12.2-7.el7rhgs.x86_64

> executed the scenario mentioned in description and conservative merged didn't happened which is expected.

> After removing file f5 and brought up the down brick, below is the files from backend

sub-vol-1
# ls /bricks/brick0/testvol_distributed-replicated_brick0/dir11/
f4  f6  f7  f8  f9
# 

sub-vol-2

# ls /bricks/brick0/testvol_distributed-replicated_brick3/dir11/
f1  f10  f2  f3
# 

Node where brick is killed and brought up

# ls /bricks/brick0/testvol_distributed-replicated_brick5/dir11/
f1  f10  f2  f3
# 

from client :

# ls
f1  f10  f2  f3  f4  f6  f7  f8  f9
# 


# gluster vol quota 23 list-obj
                  Path                   Hard-limit   Soft-limit      Files       Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------
/dir11                                          10       80%(8)          9         1           0             Yes                  Yes
# 

Changing status to verified

Comment 11 errata-xmlrpc 2018-09-04 06:29:40 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607

Note You need to log in before you can comment on or make changes to this bug.