Bug 1640148
| Summary: | Healing is not completed on Distributed-Replicated | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Vijay Avuthu <vavuthu> |
| Component: | arbiter | Assignee: | Karthik U S <ksubrahm> |
| Status: | CLOSED ERRATA | QA Contact: | Prasanth <pprakash> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rhgs-3.4 | CC: | asakthiv, jahernan, ksubrahm, nravinas, pprakash, puebele, ravishankar, rhs-bugs, rkothiya, sheggodu, storage-qa-internal |
| Target Milestone: | --- | Keywords: | Automation, ZStream |
| Target Release: | RHGS 3.5.z Batch Update 4 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-6.0-51 | Doc Type: | Bug Fix |
| Doc Text: |
Previously, there could have been a scenario where during entry heal of a directory rename, a directory in a new location could be created before deleting it from the old location, which could have led to two directories with the same gfid and few entries in pending heal state. With this update, a new volume option called cluster.use-anonymous-inode, is introduced which is ON by default on the newly created volumes with the op-version being GD_OP_VERSIONS_9_0 or higher. In this setting as part of heal, if the old location heals first and is not present in the source brick, it is best to rename it into a hidden directory inside the sink brick so that when heal is triggered in the new location, the self heal daemon renames it from this hidden directory to the new location. If new location heal is triggered first and it detects that the directory already exists in the brick, then it should skip healing the directory until it appears in the hidden directory. This volume option is OFF for older volumes created with op-version lesser than GD_OP_VERSION_9_0.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-04-29 07:20:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1786553 | ||
|
Description
Vijay Avuthu
2018-10-17 12:33:34 UTC
Healing is not able to complete because the gfid handle (symlink) for a directory is missing. Steps to reproduce in a single node setup: 0."pip install python-docx" if you don't have it. 1. Create and fuse mount a 2x (2+1) volume on /mnt/fuse_mnt. 2. Fill data on mount: python file_dir_ops.py create_deep_dirs_with_files --dir-length 2 --dir-depth 2 --max-num-of-dirs 2 --num-of-files 20 /mnt/fuse_mnt/files python file_dir_ops.py create_files -f 20 /mnt/fuse_mnt/files 3. Kill 1st data brick of each replica 4. Rename files using: python file_dir_ops.py mv /mnt/fuse_mnt/files 5. gluster volume start volname force 6. gluster volume heal volname 7. You will still see a directory and files under it not getting healed. If you look at the bricks you killed in step 3, it won't have the symlink for the directory. ------------------------------------------------------------------------------- Pranith initially had a dirty fix which solves the problem but he found some more races between janitor thread unlinking the gfid handle and posix_lookup and posix_mkdir. He and Xavi are discussing some solutions to solve this in a clean way for both AFR and EC volumes. See BZ 1636902 comment#14 and 15 for some proposed solutions. diff --git a/xlators/storage/posix/src/posix.c b/xlators/storage/posix/src/posix.c index 7bfe780bb..82f44a012 100644 --- a/xlators/storage/posix/src/posix.c +++ b/xlators/storage/posix/src/posix.c @@ -1732,7 +1732,11 @@ posix_mkdir (call_frame_t *frame, xlator_t *this, * posix_gfid_set to set the symlink to the * new dir.*/ posix_handle_unset (this, stbuf.ia_gfid, NULL); + } else if (op_ret < 0) { + MAKE_HANDLE_GFID_PATH (gfid_path, this, uuid_req, NULL); + sys_unlink(gfid_path); } + } else if (!uuid_req && frame->root->pid != GF_SERVER_PID_TRASH) { op_ret = -1; op_errno = EPERM; ------------------------------------------------------------------------------- @Pranith, feel free to assign the bug back to me if you won't be working on this. Leaving a need-info on you for this. Hit this issue, manually too while upgrading from 3.4.0-async to 3.4.1 more details refer to BZ#1643919 and https://bugzilla.redhat.com/show_bug.cgi?id=1643919#c10 *** Bug 1658870 has been marked as a duplicate of this bug. *** *** Bug 1712225 has been marked as a duplicate of this bug. *** Work on this bug will start once https://review.gluster.org/c/glusterfs/+/23937 and a subsequent patch where shd will remove stale entries is merged so moving to next BU for now. Design for https://review.gluster.org/c/glusterfs/+/23937 changed and the new patch we need for this work is https://review.gluster.org/c/glusterfs/+/24284 (In reply to Ravishankar N from comment #2) > Healing is not able to complete because the gfid handle (symlink) for a > directory is missing. Steps to reproduce in a single node setup: > > > 0."pip install python-docx" if you don't have it. > 1. Create and fuse mount a 2x (2+1) volume on /mnt/fuse_mnt. > 2. Fill data on mount: > python file_dir_ops.py create_deep_dirs_with_files --dir-length 2 > --dir-depth 2 --max-num-of-dirs 2 --num-of-files 20 /mnt/fuse_mnt/files > python file_dir_ops.py create_files -f 20 /mnt/fuse_mnt/files > 3. Kill 1st data brick of each replica > 4. Rename files using: > python file_dir_ops.py mv /mnt/fuse_mnt/files > 5. gluster volume start volname force > 6. gluster volume heal volname > 7. You will still see a directory and files under it not getting healed. If > you look at the bricks you killed in step 3, it won't have the symlink for > the directory. > > ----------------------------------------------------------------------------- > -- > Pranith initially had a dirty fix which solves the problem but he found some > more races between janitor thread unlinking the gfid handle and posix_lookup > and posix_mkdir. He and Xavi are discussing some solutions to solve this in > a clean way for both AFR and EC volumes. See BZ 1636902 comment#14 and 15 > for some proposed solutions. > > diff --git a/xlators/storage/posix/src/posix.c > b/xlators/storage/posix/src/posix.c > index 7bfe780bb..82f44a012 100644 > --- a/xlators/storage/posix/src/posix.c > +++ b/xlators/storage/posix/src/posix.c > @@ -1732,7 +1732,11 @@ posix_mkdir (call_frame_t *frame, xlator_t *this, > * posix_gfid_set to set the symlink to the > * new dir.*/ > posix_handle_unset (this, stbuf.ia_gfid, > NULL); > + } else if (op_ret < 0) { > + MAKE_HANDLE_GFID_PATH (gfid_path, this, uuid_req, NULL); > + sys_unlink(gfid_path); > } > + > } else if (!uuid_req && frame->root->pid != GF_SERVER_PID_TRASH) { > op_ret = -1; > op_errno = EPERM; > ----------------------------------------------------------------------------- > -- > @Pranith, feel free to assign the bug back to me if you won't be working on > this. Leaving a need-info on you for this. Ravi, Could you attach file_dir_ops.py script to the bz? Pranith Upasana gave the link on chat. https://github.com/gluster/glusto-tests/blob/master/glustolibs-io/shared_files/scripts/file_dir_ops.py Tested this case by running the case in comment-2 10 times and printing the number of pending heals. Before the case it was failing once in 2 times. [root@localhost-live ~]# bash testcase.sh pending-heals: 26 pending-heals: 0 pending-heals: 4 pending-heals: 0 pending-heals: 52 pending-heals: 0 pending-heals: 18 pending-heals: 0 pending-heals: 108 pending-heals: 12 With the fix it doesn't fail: [root@localhost-live ~]# bash testcase.sh pending-heals: 0 pending-heals: 0 pending-heals: 0 pending-heals: 0 pending-heals: 0 pending-heals: 0 pending-heals: 0 pending-heals: 0 pending-heals: 0 pending-heals: 0 *** Bug 1901154 has been marked as a duplicate of this bug. *** Hi Karthik, Could you please set the "Doc Type" field and fill out the "Doc Text" template with the relevant information. Thanks Amrita Thanks Karthik, LGTM. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (glusterfs bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1462 |