Bug 1482812
Summary: | [afr] split-brain observed on T files post hardlink and rename in x3 volume | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Rahul Hinduja <rhinduja> | |
Component: | replicate | Assignee: | Ravishankar N <ravishankar> | |
Status: | CLOSED ERRATA | QA Contact: | Vijay Avuthu <vavuthu> | |
Severity: | urgent | Docs Contact: | ||
Priority: | medium | |||
Version: | rhgs-3.3 | CC: | rhs-bugs, sheggodu, storage-qa-internal | |
Target Milestone: | --- | |||
Target Release: | RHGS 3.4.0 | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | rebase | |||
Fixed In Version: | glusterfs-3.12.2-1 | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1491670 (view as bug list) | Environment: | ||
Last Closed: | 2018-09-04 06:35:11 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1491670, 1496317, 1496321, 1503134 |
Description
Rahul Hinduja
2017-08-18 06:53:03 UTC
I was able to hit this issue (T files ending up in split brain) like so: 1. Create 2 x 3 volume and disable all heals: Brick1: 127.0.0.2:/home/ravi/bricks/brick1 Brick2: 127.0.0.2:/home/ravi/bricks/brick2 Brick3: 127.0.0.2:/home/ravi/bricks/brick3 Brick4: 127.0.0.2:/home/ravi/bricks/brick4 Brick5: 127.0.0.2:/home/ravi/bricks/brick5 Brick6: 127.0.0.2:/home/ravi/bricks/brick6 2. Create a file and 3 hardlinks to it from fuse mount. #tree /mnt/fuse_mnt/ /mnt/fuse_mnt/ ├── FILE ├── HLINK1 ├── HLINK3 └── HLINK7 All of these files hashed to the first dht subvol, i.e. replicate-0. 3. Kill brick4, rename HLINK1 to an appropriate name so that it gets hashed to replicate-1 and a T file is created there. 4. Likewise rename HLINK2 and HLINK3 as will, killing brick5 and brick6 respectively each time. i.e. a different brick of the 2nd replica is down each time. 5. Now enable shd and let selfheals complete. 6. File names from the mount after rename: [root@tuxpad ravi]# tree /mnt/fuse_mnt/ /mnt/fuse_mnt/ ├── FILE ├── NEW-HLINK1 ├── NEW-HLINK3-NEW └── NEW-HLINK7-NEW 7. The T files are now in split-brain: [root@tuxpad ravi]# ll /home/ravi/bricks/brick{4..6}/NEW-HLINK1 ---------T. 4 root root 0 Aug 18 12:59 /home/ravi/bricks/brick4/NEW-HLINK1 ---------T. 4 root root 0 Aug 18 12:58 /home/ravi/bricks/brick5/NEW-HLINK1 ---------T. 4 root root 0 Aug 18 12:58 /home/ravi/bricks/brick6/NEW-HLINK1 [root@tuxpad ravi]# [root@tuxpad ravi]# getfattr -d -m . -e hex /home/ravi/bricks/brick{4..6}/NEW-HLINK1 getfattr: Removing leading '/' from absolute path names # file: home/ravi/bricks/brick4/NEW-HLINK1 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a757365725f686f6d655f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.afr.testvol-client-4=0x000000010000000200000000 trusted.afr.testvol-client-5=0x000000010000000200000000 trusted.gfid=0xb4bc9ec1f7a44fa3958d82b54fc2b495 trusted.glusterfs.dht.linkto=0x74657374766f6c2d7265706c69636174652d3000 # file: home/ravi/bricks/brick5/NEW-HLINK1 security.selinux=0x73797374656d5f753a6f626a6563745f723a757365725f686f6d655f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.afr.testvol-client-3=0x000000010000000200000000 trusted.afr.testvol-client-5=0x000000010000000200000000 trusted.gfid=0xb4bc9ec1f7a44fa3958d82b54fc2b495 trusted.glusterfs.dht.linkto=0x74657374766f6c2d7265706c69636174652d3000 # file: home/ravi/bricks/brick6/NEW-HLINK1 security.selinux=0x73797374656d5f753a6f626a6563745f723a757365725f686f6d655f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.afr.testvol-client-3=0x000000010000000200000000 trusted.afr.testvol-client-4=0x000000010000000200000000 trusted.gfid=0xb4bc9ec1f7a44fa3958d82b54fc2b495 trusted.glusterfs.dht.linkto=0x74657374766f6c2d7265706c69636174652d3000 Heal-info also shows the T files to be in split-brain. Upstream patch: https://review.gluster.org/#/c/18283/1 (In reply to Ravishankar N from comment #7) > Upstream patch: https://review.gluster.org/#/c/18283/1 There is also a follow-up patch: https://review.gluster.org/#/c/18391/ (so 2 patches in total for this bug) Update: ======== Build Used : glusterfs-3.12.2-8.el7rhgs.x86_64 Scenario: 1) Create 2 * 3 distribute replicate volume and disable all heals: 2) Create a file and 3 hardlinks to it from fuse mount. All of these files hashed to the first dht subvol, i.e. replicate-0. 3) Kill brick4, rename HLINK1 to an appropriate name so that it gets hashed to replicate-1 and a T file is created there. 4) Likewise rename HLINK3 and HLINK7 as will, killing brick5 and brick6 respectively each time. i.e. a different brick of the 2nd replica is down each time. e eg : after renaming [root@dhcp35-125 ~]# tree /mnt/23/ /mnt/23/ ├── FILE ├── NEW-HLINK1 ├── NEW-HLINK3-NEW └── NEW-HLINK7-NEW 5) Now enable shd and let selfheals complete. 6) heal should complete without split-brains. All files are healed after enabling shd eg from 2nd dht subvol node : [root@dhcp35-163 ~]# ls -lrt /bricks/brick0/testvol_distributed-replicated_brick3/ total 12 ---------T. 4 root root 0 May 6 07:02 NEW-HLINK7-NEW ---------T. 4 root root 0 May 6 07:02 NEW-HLINK3-NEW ---------T. 4 root root 0 May 6 07:02 NEW-HLINK1 [root@dhcp35-163 ~]# Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607 |