Bug 1341650 - conservative merge happening on a x3 volume for a deleted file
Summary: conservative merge happening on a x3 volume for a deleted file
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: replicate
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Ravishankar N
QA Contact:
URL:
Whiteboard:
Depends On: 1341190
Blocks: 1344559 1344561
TreeView+ depends on / blocked
 
Reported: 2016-06-01 12:25 UTC by Ravishankar N
Modified: 2017-03-27 18:12 UTC (History)
2 users (show)

Fixed In Version: glusterfs-3.9.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1341190
: 1344559 1344561 (view as bug list)
Environment:
Last Closed: 2017-03-27 18:12:32 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Ravishankar N 2016-06-01 12:25:35 UTC
+++ This bug was initially created as a clone of Bug #1341190 +++

Description of problem:
=========================
When I set the object limit of a directory and bring down a brick and then exceed quota object limits and then delete an existing file, I see that a conservative merge happens for that file once the brick is brought online, instead of deleting the file from the offline brick


Version-Release number of selected component (if applicable):
=====
3.7.9-6



Steps to Reproduce:
1.created a 2x3 volume
2.enabled quota, set the hard and soft timeouts to 0
3.fuse mounted the vol
4. created a dir /dir11
5. set the quota obj-limit to 10
6. created file f{1..5}--> f1 got created on subvol-1 and remaining on sbvol2
7. brought down brick6( the 3rd brick of 2nd subvol
8. created f{6..9} using touch -->passed
9. 10th file create failed as below
touch f10
touch: cannot touch ‘f10’: Disk quota exceeded
10. now did an rm of f5
[root@dhcp35-103 dir11]# rm f5
rm: remove regular empty file ‘f5’? y
==> that means now the quota level has not exceeded limit
11. more file creates   
[root@dhcp35-103 dir11]# touch f10
[root@dhcp35-103 dir11]# touch f11
touch: cannot touch ‘f11’: Disk quota exceeded

12. bring back down brick online using start force
[root@dhcp35-103 dir11]#  ####now going to bring bricks back###

12. checked that instead of f5 getting deleted from brick6, a conserveative merge happend and f5 is created on all bricks.

==>also the quota limit has now crossed the limit from backend (heal) operation

                  Path                   Hard-limit   Soft-limit      Files       Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------

/dir11                                          10       80%(8)         10         1           0             Yes                  Yes






Getfattr details of subvol 2 :

brick4(1st of subvol2):
==>before bringing brick6 down
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

==>after bringing brick6 down and creating some files and quota limit exceeded
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

=-==>after delete of file f5
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.afr.distrepx3-client-5=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001

[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or directory


=> after created f10(which could accomdate due to deelte of f5 whih passed while f11 failed due to quota exceeded##
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000002
trusted.afr.distrepx3-client-5=0x000000000000000000000002
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

===>after bringing brick6 online using force start
[root@nchilaka-node1 glusterfs]# ####now going to bring bricks back###
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# ##brick is back online
[root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# 
[root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.bit-rot.version=0x0200000000000000574d56f500040650
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001




=======>getfattr for brick5(2nd brick of subvol2)
[root@nchilaka-node2 glusterfs]# ################ dir11 start##############
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
[root@nchilaka-node2 glusterfs]# #getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node2 glusterfs]# #########bringing down brick
[root@nchilaka-node2 glusterfs]# ps -ef|grep 4308
root      4197 27604  0 17:22 pts/0    00:00:00 grep 4308
[root@nchilaka-node2 glusterfs]# kill -9 4308
-bash: kill: (4308) - No such process
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ##deleted f5 and created some files and quota limit exceeded#
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# #getfattr of deleted file #######;getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root@nchilaka-node2 glusterfs]# ##sorry tried to delete f7 which was not even part of filsystem during that time
[root@nchilaka-node2 glusterfs]# ##retrying delete of f5
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ##deleted f5
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.afr.distrepx3-client-5=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001

[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or directory
[root@nchilaka-node2 glusterfs]# ##created f10 whih passed while f11 failed due to quota exceeded##
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000002
trusted.afr.distrepx3-client-5=0x000000000000000000000002
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node2 glusterfs]# ####now going to bring bricks back###
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ##brick is back online
[root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# 
[root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.bit-rot.version=0x0200000000000000574d56f70008f91f
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001


===>brick6 (3rd brick of subvol2) The brick which is brought down as part of test


[root@nchilaka-node4 glusterfs]# ################ dir11 start##############
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
[root@nchilaka-node4 glusterfs]# #getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node4 glusterfs]# #########bringing down brick
[root@nchilaka-node4 glusterfs]# ps -ef|grep 4308
root      4308     1  0 17:18 ?        00:00:00 /usr/sbin/glusterfsd -s 10.70.42.123 --volfile-id distrepx3.10.70.42.123.rhs-brick2-distrepx3 -p /var/lib/glusterd/vols/distrepx3/run/10.70.42.123-rhs-brick2-distrepx3.pid -S /var/run/gluster/3e4dee7a2dcfb7735da718783949470e.socket --brick-name /rhs/brick2/distrepx3 -l /var/log/glusterfs/bricks/rhs-brick2-distrepx3.log --xlator-option *-posix.glusterd-uuid=51bc4ded-1ebb-4ec3-b5a3-2b5de0b025d1 --brick-port 49162 --xlator-option distrepx3-server.listen-port=49162
root      4404 27788  0 17:22 pts/0    00:00:00 grep 4308
[root@nchilaka-node4 glusterfs]# kill -9 4308
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ##deleted f5 and created some files and quota limit exceeded#
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# #getfattr of deleted file #######;getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root@nchilaka-node4 glusterfs]# ##sorry tried to delete f7 which was not even part of filsystem during that time
[root@nchilaka-node4 glusterfs]# ##retrying delete of f5
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ##deleted f5
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root@nchilaka-node4 glusterfs]# ##created f10 whih passed while f11 failed due to quota exceeded##
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root@nchilaka-node4 glusterfs]# ####now going to bring bricks back###
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ##brick is back online
[root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.distrepx3-client-3=0x000000000000000000000000
trusted.afr.distrepx3-client-4=0x000000000000000000000000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root@nchilaka-node4 glusterfs]# 
[root@nchilaka-node4 glusterfs]# #############creating new vol testnew
[root@nchilaka-node4 glusterfs]# rm -rf /rhs/brick1/testnew
[root@nchilaka-node4 glusterfs]# 




===>other cmd o/p
[root@nchilaka-node1 ~]# gluster v status distrepx3
Status of volume: distrepx3
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.43.22:/rhs/brick1/distrepx3     49160     0          Y       5130 
Brick 10.70.42.188:/rhs/brick1/distrepx3    49158     0          Y       31389
Brick 10.70.42.123:/rhs/brick1/distrepx3    49161     0          Y       2296 
Brick 10.70.43.22:/rhs/brick2/distrepx3     49161     0          Y       5149 
Brick 10.70.42.188:/rhs/brick2/distrepx3    49159     0          Y       31408
Brick 10.70.42.123:/rhs/brick2/distrepx3    49162     0          Y       4776 
NFS Server on localhost                     2049      0          Y       11839
Self-heal Daemon on localhost               N/A       N/A        Y       11847
Quota Daemon on localhost                   N/A       N/A        Y       11855
NFS Server on 10.70.42.123                  2049      0          Y       4796 
Self-heal Daemon on 10.70.42.123            N/A       N/A        Y       4804 
Quota Daemon on 10.70.42.123                N/A       N/A        Y       4813 
NFS Server on 10.70.42.204                  2049      0          Y       27323
Self-heal Daemon on 10.70.42.204            N/A       N/A        Y       27331
Quota Daemon on 10.70.42.204                N/A       N/A        Y       27339
NFS Server on 10.70.42.188                  2049      0          Y       4573 
Self-heal Daemon on 10.70.42.188            N/A       N/A        Y       4581 
Quota Daemon on 10.70.42.188                N/A       N/A        Y       4590 
 
Task Status of Volume distrepx3
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@nchilaka-node1 ~]# gluster v quota distrepx3 list-obj
                  Path                   Hard-limit   Soft-limit      Files       Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------
/dir1                                           35       80%(28)         90         1           0             Yes                  Yes
/dir2                                          200       80%(160)        300         1           0             Yes                  Yes
/dir3                                         1100       80%(880)       1210         1           0             Yes                  Yes
/dir4                                           50       80%(40)         40         1           9             Yes                   No
/dir5                                          100       80%(80)        199         1           0             Yes                  Yes
/dir6                                          100       80%(80)         99         1           0             Yes                  Yes
/dir7                                           10       80%(8)         10         1           0             Yes                  Yes
/dir8                                           10       80%(8)          9         1           0             Yes                  Yes
/dir10                                          10       80%(8)          9         1           0             Yes                  Yes
/dir11                                          10       80%(8)         10         1           0             Yes                  Yes
[root@nchilaka-node1 ~]# 
[root@nchilaka-node1 ~]# 
[root@nchilaka-node1 ~]# gluster v heal distrepx3 info
Brick 10.70.43.22:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.188:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.123:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.43.22:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.188:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.123:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0


 
Volume Name: distrepx3
Type: Distributed-Replicate
Volume ID: 353f80dc-aa40-468d-b6ae-8604b79e3397
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.43.22:/rhs/brick1/distrepx3
Brick2: 10.70.42.188:/rhs/brick1/distrepx3
Brick3: 10.70.42.123:/rhs/brick1/distrepx3
Brick4: 10.70.43.22:/rhs/brick2/distrepx3
Brick5: 10.70.42.188:/rhs/brick2/distrepx3
Brick6: 10.70.42.123:/rhs/brick2/distrepx3
Options Reconfigured:
features.soft-timeout: 0
features.hard-timeout: 0
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on


--- Additional comment from Ravishankar N on 2016-06-01 05:30:31 EDT ---

RCA:
I was able to re-create the issue on a 1x3 plain replica (RHGS 3.1.3) with the same steps given in the description. If one of the bricks of a replica is down, AFR marks dirty xattrs for even entry transactions ('touch f10' in this case) in the pre-op. In the post-op, since both the up bricks return EDQUOT, we consider it a symmetric error and don't do the post-op, retaining the dirty xattr for the parent directory. When self-heal is triggered when all bricks are up, it considers directories with dirty xattr as a candidate for conservative merge. Hence the deleted file is re created on the source bricks instead of being deleted from the sink.

Comment 1 Vijay Bellur 2016-06-01 12:42:44 UTC
REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as symmetric errors) posted (#1) for review on master by Ravishankar N (ravishankar)

Comment 2 Vijay Bellur 2016-06-01 12:45:08 UTC
REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as symmetric errors) posted (#2) for review on master by Ravishankar N (ravishankar)

Comment 3 Vijay Bellur 2016-06-06 11:22:48 UTC
REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as symmetric errors) posted (#3) for review on master by Ravishankar N (ravishankar)

Comment 4 Vijay Bellur 2016-06-10 11:31:58 UTC
REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as symmetric errors) posted (#4) for review on master by Ravishankar N (ravishankar)

Comment 5 Vijay Bellur 2016-06-13 05:58:52 UTC
COMMIT: http://review.gluster.org/14604 committed in master by Atin Mukherjee (amukherj) 
------
commit 8c717491413295a7045ebf9a2b1df630b7cbac76
Author: Ravishankar N <ravishankar>
Date:   Wed Jun 1 17:57:41 2016 +0530

    afr: Consider ENOSPC and EDQUOT as symmetric errors
    
    Problem:
    Since commit 8eaa3506ead4f11b81b146a9e56575c79f3aad7b, in replica 3, if a
    brick is down and a create fails on the other 2 brick with EDQUOT, we consider
    it an unsymmetric error and hence do not do post-op. So the dirty xattr
    remains set on the parent dir, leading to conservative merges during heal when
    all bricks are up. i.e. a file deleted on the source might re-appear after heal.
    
    Fix:
    Consider ENOSPC and  EDQUOT as symmetric errors since there is no
    possibility of partial inode or entry modification operations possible when
    quota is enabled. IOW, if quota reports EDQUOT, the no. of bytes written
    (or not written) will be the same on all bricks of the replica.
    Likewise, the entry operation (create, mkdir...) will either succeed or
    not succeed on all bricks.
    
    Change-Id: Iacb1108e9ef4a918e36242fb4a957455133744e9
    BUG: 1341650
    Signed-off-by: Ravishankar N <ravishankar>
    Reviewed-on: http://review.gluster.org/14604
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Atin Mukherjee <amukherj>
    Tested-by: Atin Mukherjee <amukherj>

Comment 6 Shyamsundar 2017-03-27 18:12:32 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report.

glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.