+++ This bug was initially created as a clone of Bug #1341190 +++ Description of problem: ========================= When I set the object limit of a directory and bring down a brick and then exceed quota object limits and then delete an existing file, I see that a conservative merge happens for that file once the brick is brought online, instead of deleting the file from the offline brick Version-Release number of selected component (if applicable): ===== 3.7.9-6 Steps to Reproduce: 1.created a 2x3 volume 2.enabled quota, set the hard and soft timeouts to 0 3.fuse mounted the vol 4. created a dir /dir11 5. set the quota obj-limit to 10 6. created file f{1..5}--> f1 got created on subvol-1 and remaining on sbvol2 7. brought down brick6( the 3rd brick of 2nd subvol 8. created f{6..9} using touch -->passed 9. 10th file create failed as below touch f10 touch: cannot touch ‘f10’: Disk quota exceeded 10. now did an rm of f5 [root@dhcp35-103 dir11]# rm f5 rm: remove regular empty file ‘f5’? y ==> that means now the quota level has not exceeded limit 11. more file creates [root@dhcp35-103 dir11]# touch f10 [root@dhcp35-103 dir11]# touch f11 touch: cannot touch ‘f11’: Disk quota exceeded 12. bring back down brick online using start force [root@dhcp35-103 dir11]# ####now going to bring bricks back### 12. checked that instead of f5 getting deleted from brick6, a conserveative merge happend and f5 is created on all bricks. ==>also the quota limit has now crossed the limit from backend (heal) operation Path Hard-limit Soft-limit Files Dirs Available Soft-limit exceeded? Hard-limit exceeded? ----------------------------------------------------------------------------------------------------------------------------------------------- /dir11 10 80%(8) 10 1 0 Yes Yes Getfattr details of subvol 2 : brick4(1st of subvol2): ==>before bringing brick6 down [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 ==>after bringing brick6 down and creating some files and quota limit exceeded [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000001 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 =-==>after delete of file f5 [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000001 trusted.afr.distrepx3-client-5=0x000000000000000000000001 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001 [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or directory => after created f10(which could accomdate due to deelte of f5 whih passed while f11 failed due to quota exceeded## [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000002 trusted.afr.distrepx3-client-5=0x000000000000000000000002 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 ===>after bringing brick6 online using force start [root@nchilaka-node1 glusterfs]# ####now going to bring bricks back### [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# ##brick is back online [root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.afr.distrepx3-client-5=0x000000000000000000000000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001 [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.bit-rot.version=0x0200000000000000574d56f500040650 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 =======>getfattr for brick5(2nd brick of subvol2) [root@nchilaka-node2 glusterfs]# ################ dir11 start############## [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 [root@nchilaka-node2 glusterfs]# #getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node2 glusterfs]# #########bringing down brick [root@nchilaka-node2 glusterfs]# ps -ef|grep 4308 root 4197 27604 0 17:22 pts/0 00:00:00 grep 4308 [root@nchilaka-node2 glusterfs]# kill -9 4308 -bash: kill: (4308) - No such process [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ##deleted f5 and created some files and quota limit exceeded# [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000001 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# #getfattr of deleted file #######;getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 [root@nchilaka-node2 glusterfs]# ##sorry tried to delete f7 which was not even part of filsystem during that time [root@nchilaka-node2 glusterfs]# ##retrying delete of f5 [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ##deleted f5 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000001 trusted.afr.distrepx3-client-5=0x000000000000000000000001 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or directory [root@nchilaka-node2 glusterfs]# ##created f10 whih passed while f11 failed due to quota exceeded## [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000002 trusted.afr.distrepx3-client-5=0x000000000000000000000002 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node2 glusterfs]# ####now going to bring bricks back### [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ##brick is back online [root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.afr.distrepx3-client-5=0x000000000000000000000000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.bit-rot.version=0x0200000000000000574d56f70008f91f trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 ===>brick6 (3rd brick of subvol2) The brick which is brought down as part of test [root@nchilaka-node4 glusterfs]# ################ dir11 start############## [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 [root@nchilaka-node4 glusterfs]# #getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node4 glusterfs]# #########bringing down brick [root@nchilaka-node4 glusterfs]# ps -ef|grep 4308 root 4308 1 0 17:18 ? 00:00:00 /usr/sbin/glusterfsd -s 10.70.42.123 --volfile-id distrepx3.10.70.42.123.rhs-brick2-distrepx3 -p /var/lib/glusterd/vols/distrepx3/run/10.70.42.123-rhs-brick2-distrepx3.pid -S /var/run/gluster/3e4dee7a2dcfb7735da718783949470e.socket --brick-name /rhs/brick2/distrepx3 -l /var/log/glusterfs/bricks/rhs-brick2-distrepx3.log --xlator-option *-posix.glusterd-uuid=51bc4ded-1ebb-4ec3-b5a3-2b5de0b025d1 --brick-port 49162 --xlator-option distrepx3-server.listen-port=49162 root 4404 27788 0 17:22 pts/0 00:00:00 grep 4308 [root@nchilaka-node4 glusterfs]# kill -9 4308 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ##deleted f5 and created some files and quota limit exceeded# [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# #getfattr of deleted file #######;getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 [root@nchilaka-node4 glusterfs]# ##sorry tried to delete f7 which was not even part of filsystem during that time [root@nchilaka-node4 glusterfs]# ##retrying delete of f5 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ##deleted f5 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 [root@nchilaka-node4 glusterfs]# ##created f10 whih passed while f11 failed due to quota exceeded## [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node4 glusterfs]# ####now going to bring bricks back### [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ##brick is back online [root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.distrepx3-client-3=0x000000000000000000000000 trusted.afr.distrepx3-client-4=0x000000000000000000000000 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# #############creating new vol testnew [root@nchilaka-node4 glusterfs]# rm -rf /rhs/brick1/testnew [root@nchilaka-node4 glusterfs]# ===>other cmd o/p [root@nchilaka-node1 ~]# gluster v status distrepx3 Status of volume: distrepx3 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.43.22:/rhs/brick1/distrepx3 49160 0 Y 5130 Brick 10.70.42.188:/rhs/brick1/distrepx3 49158 0 Y 31389 Brick 10.70.42.123:/rhs/brick1/distrepx3 49161 0 Y 2296 Brick 10.70.43.22:/rhs/brick2/distrepx3 49161 0 Y 5149 Brick 10.70.42.188:/rhs/brick2/distrepx3 49159 0 Y 31408 Brick 10.70.42.123:/rhs/brick2/distrepx3 49162 0 Y 4776 NFS Server on localhost 2049 0 Y 11839 Self-heal Daemon on localhost N/A N/A Y 11847 Quota Daemon on localhost N/A N/A Y 11855 NFS Server on 10.70.42.123 2049 0 Y 4796 Self-heal Daemon on 10.70.42.123 N/A N/A Y 4804 Quota Daemon on 10.70.42.123 N/A N/A Y 4813 NFS Server on 10.70.42.204 2049 0 Y 27323 Self-heal Daemon on 10.70.42.204 N/A N/A Y 27331 Quota Daemon on 10.70.42.204 N/A N/A Y 27339 NFS Server on 10.70.42.188 2049 0 Y 4573 Self-heal Daemon on 10.70.42.188 N/A N/A Y 4581 Quota Daemon on 10.70.42.188 N/A N/A Y 4590 Task Status of Volume distrepx3 ------------------------------------------------------------------------------ There are no active volume tasks [root@nchilaka-node1 ~]# gluster v quota distrepx3 list-obj Path Hard-limit Soft-limit Files Dirs Available Soft-limit exceeded? Hard-limit exceeded? ----------------------------------------------------------------------------------------------------------------------------------------------- /dir1 35 80%(28) 90 1 0 Yes Yes /dir2 200 80%(160) 300 1 0 Yes Yes /dir3 1100 80%(880) 1210 1 0 Yes Yes /dir4 50 80%(40) 40 1 9 Yes No /dir5 100 80%(80) 199 1 0 Yes Yes /dir6 100 80%(80) 99 1 0 Yes Yes /dir7 10 80%(8) 10 1 0 Yes Yes /dir8 10 80%(8) 9 1 0 Yes Yes /dir10 10 80%(8) 9 1 0 Yes Yes /dir11 10 80%(8) 10 1 0 Yes Yes [root@nchilaka-node1 ~]# [root@nchilaka-node1 ~]# [root@nchilaka-node1 ~]# gluster v heal distrepx3 info Brick 10.70.43.22:/rhs/brick1/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.42.188:/rhs/brick1/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.42.123:/rhs/brick1/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.43.22:/rhs/brick2/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.42.188:/rhs/brick2/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.42.123:/rhs/brick2/distrepx3 Status: Connected Number of entries: 0 Volume Name: distrepx3 Type: Distributed-Replicate Volume ID: 353f80dc-aa40-468d-b6ae-8604b79e3397 Status: Started Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.70.43.22:/rhs/brick1/distrepx3 Brick2: 10.70.42.188:/rhs/brick1/distrepx3 Brick3: 10.70.42.123:/rhs/brick1/distrepx3 Brick4: 10.70.43.22:/rhs/brick2/distrepx3 Brick5: 10.70.42.188:/rhs/brick2/distrepx3 Brick6: 10.70.42.123:/rhs/brick2/distrepx3 Options Reconfigured: features.soft-timeout: 0 features.hard-timeout: 0 features.quota-deem-statfs: on features.inode-quota: on features.quota: on performance.readdir-ahead: on --- Additional comment from Ravishankar N on 2016-06-01 05:30:31 EDT --- RCA: I was able to re-create the issue on a 1x3 plain replica (RHGS 3.1.3) with the same steps given in the description. If one of the bricks of a replica is down, AFR marks dirty xattrs for even entry transactions ('touch f10' in this case) in the pre-op. In the post-op, since both the up bricks return EDQUOT, we consider it a symmetric error and don't do the post-op, retaining the dirty xattr for the parent directory. When self-heal is triggered when all bricks are up, it considers directories with dirty xattr as a candidate for conservative merge. Hence the deleted file is re created on the source bricks instead of being deleted from the sink.
REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as symmetric errors) posted (#1) for review on master by Ravishankar N (ravishankar)
REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as symmetric errors) posted (#2) for review on master by Ravishankar N (ravishankar)
REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as symmetric errors) posted (#3) for review on master by Ravishankar N (ravishankar)
REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as symmetric errors) posted (#4) for review on master by Ravishankar N (ravishankar)
COMMIT: http://review.gluster.org/14604 committed in master by Atin Mukherjee (amukherj) ------ commit 8c717491413295a7045ebf9a2b1df630b7cbac76 Author: Ravishankar N <ravishankar> Date: Wed Jun 1 17:57:41 2016 +0530 afr: Consider ENOSPC and EDQUOT as symmetric errors Problem: Since commit 8eaa3506ead4f11b81b146a9e56575c79f3aad7b, in replica 3, if a brick is down and a create fails on the other 2 brick with EDQUOT, we consider it an unsymmetric error and hence do not do post-op. So the dirty xattr remains set on the parent dir, leading to conservative merges during heal when all bricks are up. i.e. a file deleted on the source might re-appear after heal. Fix: Consider ENOSPC and EDQUOT as symmetric errors since there is no possibility of partial inode or entry modification operations possible when quota is enabled. IOW, if quota reports EDQUOT, the no. of bytes written (or not written) will be the same on all bricks of the replica. Likewise, the entry operation (create, mkdir...) will either succeed or not succeed on all bricks. Change-Id: Iacb1108e9ef4a918e36242fb4a957455133744e9 BUG: 1341650 Signed-off-by: Ravishankar N <ravishankar> Reviewed-on: http://review.gluster.org/14604 Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> Tested-by: Atin Mukherjee <amukherj>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report. glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html [2] https://www.gluster.org/pipermail/gluster-users/