Bug 1341190
| Summary: | conservative merge happening on a x3 volume for a deleted file | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> | |
| Component: | replicate | Assignee: | Ravishankar N <ravishankar> | |
| Status: | CLOSED ERRATA | QA Contact: | Vijay Avuthu <vavuthu> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | medium | |||
| Version: | rhgs-3.1 | CC: | rhinduja, rhs-bugs, sheggodu | |
| Target Milestone: | --- | Keywords: | ZStream | |
| Target Release: | RHGS 3.4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | rebase | |||
| Fixed In Version: | glusterfs-3.12.2-1 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1341650 (view as bug list) | Environment: | ||
| Last Closed: | 2018-09-04 06:29:40 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1341650, 1344559, 1344561, 1503134 | |||
raised this bug as part of qa on_qa validation of 1332199:Self Heal fails on a replica3 volume with 'disk quota exceeded' Update: ========== Build Used : glusterfs-3.12.2-7.el7rhgs.x86_64 > executed the scenario mentioned in description and conservative merged didn't happened which is expected. > After removing file f5 and brought up the down brick, below is the files from backend sub-vol-1 # ls /bricks/brick0/testvol_distributed-replicated_brick0/dir11/ f4 f6 f7 f8 f9 # sub-vol-2 # ls /bricks/brick0/testvol_distributed-replicated_brick3/dir11/ f1 f10 f2 f3 # Node where brick is killed and brought up # ls /bricks/brick0/testvol_distributed-replicated_brick5/dir11/ f1 f10 f2 f3 # from client : # ls f1 f10 f2 f3 f4 f6 f7 f8 f9 # # gluster vol quota 23 list-obj Path Hard-limit Soft-limit Files Dirs Available Soft-limit exceeded? Hard-limit exceeded? ----------------------------------------------------------------------------------------------------------------------------------------------- /dir11 10 80%(8) 9 1 0 Yes Yes # Changing status to verified Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607 |
Description of problem: ========================= When I set the object limit of a directory and bring down a brick and then exceed quota object limits and then delete an existing file, I see that a conservative merge happens for that file once the brick is brought online, instead of deleting the file from the offline brick Version-Release number of selected component (if applicable): ===== 3.7.9-6 Steps to Reproduce: 1.created a 2x3 volume 2.enabled quota, set the hard and soft timeouts to 0 3.fuse mounted the vol 4. created a dir /dir11 5. set the quota obj-limit to 10 6. created file f{1..5}--> f1 got created on subvol-1 and remaining on sbvol2 7. brought down brick6( the 3rd brick of 2nd subvol 8. created f{6..9} using touch -->passed 9. 10th file create failed as below touch f10 touch: cannot touch ‘f10’: Disk quota exceeded 10. now did an rm of f5 [root@dhcp35-103 dir11]# rm f5 rm: remove regular empty file ‘f5’? y ==> that means now the quota level has not exceeded limit 11. more file creates [root@dhcp35-103 dir11]# touch f10 [root@dhcp35-103 dir11]# touch f11 touch: cannot touch ‘f11’: Disk quota exceeded 12. bring back down brick online using start force [root@dhcp35-103 dir11]# ####now going to bring bricks back### 12. checked that instead of f5 getting deleted from brick6, a conserveative merge happend and f5 is created on all bricks. ==>also the quota limit has now crossed the limit from backend (heal) operation Path Hard-limit Soft-limit Files Dirs Available Soft-limit exceeded? Hard-limit exceeded? ----------------------------------------------------------------------------------------------------------------------------------------------- /dir11 10 80%(8) 10 1 0 Yes Yes Getfattr details of subvol 2 : brick4(1st of subvol2): ==>before bringing brick6 down [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 ==>after bringing brick6 down and creating some files and quota limit exceeded [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000001 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 =-==>after delete of file f5 [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000001 trusted.afr.distrepx3-client-5=0x000000000000000000000001 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001 [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or directory => after created f10(which could accomdate due to deelte of f5 whih passed while f11 failed due to quota exceeded## [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000002 trusted.afr.distrepx3-client-5=0x000000000000000000000002 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 ===>after bringing brick6 online using force start [root@nchilaka-node1 glusterfs]# ####now going to bring bricks back### [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# ##brick is back online [root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# [root@nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.afr.distrepx3-client-5=0x000000000000000000000000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001 [root@nchilaka-node1 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.bit-rot.version=0x0200000000000000574d56f500040650 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 =======>getfattr for brick5(2nd brick of subvol2) [root@nchilaka-node2 glusterfs]# ################ dir11 start############## [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 [root@nchilaka-node2 glusterfs]# #getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node2 glusterfs]# #########bringing down brick [root@nchilaka-node2 glusterfs]# ps -ef|grep 4308 root 4197 27604 0 17:22 pts/0 00:00:00 grep 4308 [root@nchilaka-node2 glusterfs]# kill -9 4308 -bash: kill: (4308) - No such process [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ##deleted f5 and created some files and quota limit exceeded# [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000001 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# #getfattr of deleted file #######;getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 [root@nchilaka-node2 glusterfs]# ##sorry tried to delete f7 which was not even part of filsystem during that time [root@nchilaka-node2 glusterfs]# ##retrying delete of f5 [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ##deleted f5 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000001 trusted.afr.distrepx3-client-5=0x000000000000000000000001 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or directory [root@nchilaka-node2 glusterfs]# ##created f10 whih passed while f11 failed due to quota exceeded## [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000002 trusted.afr.distrepx3-client-5=0x000000000000000000000002 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node2 glusterfs]# ####now going to bring bricks back### [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ##brick is back online [root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# [root@nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.afr.distrepx3-client-5=0x000000000000000000000000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001 [root@nchilaka-node2 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.bit-rot.version=0x0200000000000000574d56f70008f91f trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 ===>brick6 (3rd brick of subvol2) The brick which is brought down as part of test [root@nchilaka-node4 glusterfs]# ################ dir11 start############## [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 [root@nchilaka-node4 glusterfs]# #getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node4 glusterfs]# #########bringing down brick [root@nchilaka-node4 glusterfs]# ps -ef|grep 4308 root 4308 1 0 17:18 ? 00:00:00 /usr/sbin/glusterfsd -s 10.70.42.123 --volfile-id distrepx3.10.70.42.123.rhs-brick2-distrepx3 -p /var/lib/glusterd/vols/distrepx3/run/10.70.42.123-rhs-brick2-distrepx3.pid -S /var/run/gluster/3e4dee7a2dcfb7735da718783949470e.socket --brick-name /rhs/brick2/distrepx3 -l /var/log/glusterfs/bricks/rhs-brick2-distrepx3.log --xlator-option *-posix.glusterd-uuid=51bc4ded-1ebb-4ec3-b5a3-2b5de0b025d1 --brick-port 49162 --xlator-option distrepx3-server.listen-port=49162 root 4404 27788 0 17:22 pts/0 00:00:00 grep 4308 [root@nchilaka-node4 glusterfs]# kill -9 4308 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ##deleted f5 and created some files and quota limit exceeded# [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# #getfattr of deleted file #######;getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 [root@nchilaka-node4 glusterfs]# ##sorry tried to delete f7 which was not even part of filsystem during that time [root@nchilaka-node4 glusterfs]# ##retrying delete of f5 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ##deleted f5 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 [root@nchilaka-node4 glusterfs]# ##created f10 whih passed while f11 failed due to quota exceeded## [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001 [root@nchilaka-node4 glusterfs]# ####now going to bring bricks back### [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ##brick is back online [root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f4 -rw-r--r--. 2 root root 0 May 31 17:04 f6 -rw-r--r--. 2 root root 0 May 31 17:04 f7 -rw-r--r--. 2 root root 0 May 31 17:04 f8 -rw-r--r--. 2 root root 0 May 31 17:04 f9 [root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/ total 0 -rw-r--r--. 2 root root 0 May 31 17:02 f1 -rw-r--r--. 2 root root 0 May 31 17:07 f10 -rw-r--r--. 2 root root 0 May 31 17:02 f2 -rw-r--r--. 2 root root 0 May 31 17:02 f3 -rw-r--r--. 2 root root 0 May 31 17:02 f5 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001 [root@nchilaka-node4 glusterfs]# getfattr -d -m . -e hex /rhs/brick2/dist*/dir11/f5 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/distrepx3/dir11/f5 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.distrepx3-client-3=0x000000000000000000000000 trusted.afr.distrepx3-client-4=0x000000000000000000000000 trusted.gfid=0x7279779380764113be45950ae7484d06 trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001 trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001 [root@nchilaka-node4 glusterfs]# [root@nchilaka-node4 glusterfs]# #############creating new vol testnew [root@nchilaka-node4 glusterfs]# rm -rf /rhs/brick1/testnew [root@nchilaka-node4 glusterfs]# ===>other cmd o/p [root@nchilaka-node1 ~]# gluster v status distrepx3 Status of volume: distrepx3 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.43.22:/rhs/brick1/distrepx3 49160 0 Y 5130 Brick 10.70.42.188:/rhs/brick1/distrepx3 49158 0 Y 31389 Brick 10.70.42.123:/rhs/brick1/distrepx3 49161 0 Y 2296 Brick 10.70.43.22:/rhs/brick2/distrepx3 49161 0 Y 5149 Brick 10.70.42.188:/rhs/brick2/distrepx3 49159 0 Y 31408 Brick 10.70.42.123:/rhs/brick2/distrepx3 49162 0 Y 4776 NFS Server on localhost 2049 0 Y 11839 Self-heal Daemon on localhost N/A N/A Y 11847 Quota Daemon on localhost N/A N/A Y 11855 NFS Server on 10.70.42.123 2049 0 Y 4796 Self-heal Daemon on 10.70.42.123 N/A N/A Y 4804 Quota Daemon on 10.70.42.123 N/A N/A Y 4813 NFS Server on 10.70.42.204 2049 0 Y 27323 Self-heal Daemon on 10.70.42.204 N/A N/A Y 27331 Quota Daemon on 10.70.42.204 N/A N/A Y 27339 NFS Server on 10.70.42.188 2049 0 Y 4573 Self-heal Daemon on 10.70.42.188 N/A N/A Y 4581 Quota Daemon on 10.70.42.188 N/A N/A Y 4590 Task Status of Volume distrepx3 ------------------------------------------------------------------------------ There are no active volume tasks [root@nchilaka-node1 ~]# gluster v quota distrepx3 list-obj Path Hard-limit Soft-limit Files Dirs Available Soft-limit exceeded? Hard-limit exceeded? ----------------------------------------------------------------------------------------------------------------------------------------------- /dir1 35 80%(28) 90 1 0 Yes Yes /dir2 200 80%(160) 300 1 0 Yes Yes /dir3 1100 80%(880) 1210 1 0 Yes Yes /dir4 50 80%(40) 40 1 9 Yes No /dir5 100 80%(80) 199 1 0 Yes Yes /dir6 100 80%(80) 99 1 0 Yes Yes /dir7 10 80%(8) 10 1 0 Yes Yes /dir8 10 80%(8) 9 1 0 Yes Yes /dir10 10 80%(8) 9 1 0 Yes Yes /dir11 10 80%(8) 10 1 0 Yes Yes [root@nchilaka-node1 ~]# [root@nchilaka-node1 ~]# [root@nchilaka-node1 ~]# gluster v heal distrepx3 info Brick 10.70.43.22:/rhs/brick1/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.42.188:/rhs/brick1/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.42.123:/rhs/brick1/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.43.22:/rhs/brick2/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.42.188:/rhs/brick2/distrepx3 Status: Connected Number of entries: 0 Brick 10.70.42.123:/rhs/brick2/distrepx3 Status: Connected Number of entries: 0 Volume Name: distrepx3 Type: Distributed-Replicate Volume ID: 353f80dc-aa40-468d-b6ae-8604b79e3397 Status: Started Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.70.43.22:/rhs/brick1/distrepx3 Brick2: 10.70.42.188:/rhs/brick1/distrepx3 Brick3: 10.70.42.123:/rhs/brick1/distrepx3 Brick4: 10.70.43.22:/rhs/brick2/distrepx3 Brick5: 10.70.42.188:/rhs/brick2/distrepx3 Brick6: 10.70.42.123:/rhs/brick2/distrepx3 Options Reconfigured: features.soft-timeout: 0 features.hard-timeout: 0 features.quota-deem-statfs: on features.inode-quota: on features.quota: on performance.readdir-ahead: on