Bug 1609243
| Summary: | Able to mount dir which is in gfid split-brain and even files creation are successful in split brain dir | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Vijay Avuthu <vavuthu> |
| Component: | fuse | Assignee: | Karthik U S <ksubrahm> |
| Status: | CLOSED WONTFIX | QA Contact: | Rahul Hinduja <rhinduja> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | rhgs-3.4 | CC: | rhs-bugs, storage-qa-internal, vavuthu, vdas |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-09 11:33:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Vijay Avuthu
2018-07-27 11:11:45 UTC
Moving this bug out of 3.4.0 as this doesn't meet blocker criteria. This can be re-proposed for 3.4.0 if required. Can this be tried with replica 3 volume? A replica 2 volume type is not supported officially. Hi I am able to recreate the issue in 1X3 volume,
1. Create 1X3 volume.
2. Change quorum options to create a split-brain directory in replica 3 set.
3. Split-brain was create successfully
# gluster v heal vol_36411cb4f9b145b165212aeaf0ca2588 info
Brick 10.70.47.7:/var/lib/heketi/mounts/vg_1a2cebdd439fca0eb9d5197d6a6ca504/brick_25836af43f1427d9fe24c06feebbb1c7/brick
/ - Is in split-brain
Status: Connected
Number of entries: 1
Brick 10.70.47.108:/var/lib/heketi/mounts/vg_690b7b8be089c66b07c1259811ef6dbc/brick_04c52fc4af911b40730665ef9203304a/brick
/ - Is in split-brain
Status: Connected
Number of entries: 1
Brick 10.70.46.206:/var/lib/heketi/mounts/vg_bb4c74213d62f197a5baed1abad3df73/brick_d2469a52e6c4f94556140cec1de5582e/brick
/ - Is in split-brain
Status: Connected
Number of entries: 1
4. Change volume quorum option back to auto
4. Mount the directory to a client
# mount -t glusterfs 10.70.47.108:vol_36411cb4f9b145b165212aeaf0ca2588/dir1 /mnt/split/
Mount was successful.
5. Write files from mount point
# touch f{1..10}
# ls
f1 f10 f2 f3 f4 f5 f6 f7 f8 f9
# echo "Hi" >>f1
lit]# cat f1
Hi
Expected Result:
Shouldn't mount the dir which is in split-brain
Gluster fuse-subdir mount is a new feature (GA in 3.4), and the issue was discovered in 3.4 build. Issue is reproducible in latest build # rpm -qa | grep gluster python2-gluster-3.12.2-29.el7rhgs.x86_64 glusterfs-debuginfo-3.12.2-29.el7rhgs.x86_64 glusterfs-libs-3.12.2-29.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-29.el7rhgs.x86_64 glusterfs-3.12.2-29.el7rhgs.x86_64 Was trying to understand what to do in this case. Technically, there is no way for a client to know that there is a GFID mismatch in this scenario, mainly because client sees the GFID of this directory as 0x01 (ie, root). Server would know the GFID, but it won't have visibility into what is the 'correct' GFID for the directory, as it won't have the cluster view. I would like to understand from PM (or any others) about what should be the right behavior in this scenario ? If the right step is failing the mount by stating the file is not in correct state, then we need to make sure we change design etc. The quick and dirty fix is handling this scenario in mount.glusterfs script by checking the directory in bigger volume as temp mount first, before continuing the actual mount with subdir. While this would be good enough to handle 99% of the cases, if one uses `glusterfs` command directly to mount subdirectory, then this would still be an issue. For now, I will wait for some discussion on this here, and then we can take decision on this. |