Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: ----------------------- With RHHI-V use case, gluster replica 3 volume is used to storage VM images. RHEL 7.6 VM is created on such a setup with its root disk image on the gluster replica 3 sharded volume. Post that, template of the VM is created as a thin template(qcow2 based). Then tried creating many thin disk VMs out of that template and ran some workload on those VMs. After sometime, stopped all the VMs and deleted the same. After sometime, there was always a pending heal on that volume which finally turned out to be gfid mismatch Version-Release number of selected component (if applicable): ------------------------------------------------------------- RHGS 3.4.4 ( glusters-3.12.2-47.el7rhgs ) How reproducible: ----------------- Always Steps to Reproduce: ------------------- 1. Create RHHI-V setup 2. Use replica 3 sharded volume for storing VM images 3. Create RHEL 7 VM with its base image on this fuse mounted volume 4. Create a thin template of this VM ( qcow2 ) 5. Create many thin disk VMs out of this template. Thin disk in this context means that the base image of the template is hardlinked and overlay file is created for new writes. 6. Run some workload on these VMs which creates some files 7. Delete all the VMs along with its disks Actual results: ----------------- gfid mismatch observed Expected results: ----------------- no gfid mismatch seen Additional info:
Here is the snip from glustershd.log <snip> [2019-05-02 10:03:44.007380] E [MSGID: 108008] [afr-self-heal-common.c:352:afr_gfid_split_brain_source] 0-vmstore-replicate-0: Gfid mismatch detected for <gfid:be318638-e8a0-4c6d-977d-7a937aa84806>/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589>, 90bf5788-dca8-45e9-ab4f-7e4911f5e6f3 on vmstore-client-2 and 4fdb0da1-4cbe-4c1b-856c-b44158a1d276 on vmstore-client-1. [2019-05-02 10:03:44.009542] W [MSGID: 108027] [afr-common.c:2279:afr_attempt_readsubvol_set] 0-vmstore-replicate-0: no read subvols for (null) [2019-05-02 10:03:44.010892] E [MSGID: 108008] [afr-self-heal-common.c:352:afr_gfid_split_brain_source] 0-vmstore-replicate-0: Gfid mismatch detected for <gfid:be318638-e8a0-4c6d-977d-7a937aa84806>/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589>, 90bf5788-dca8-45e9-ab4f-7e4911f5e6f3 on vmstore-client-2 and 4fdb0da1-4cbe-4c1b-856c-b44158a1d276 on vmstore-client-0. [2019-05 </snip>
Snip from the fuse mount log <snip> [2019-04-29 09:35:19.940241] I [MSGID: 108026] [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-vmstore-replicate-0: performing entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806 [2019-04-29 09:35:19.944454] E [MSGID: 108008] [afr-self-heal-common.c:352:afr_gfid_split_brain_source] 0-vmstore-replicate-0: Gfid mismatch detected for <gfid:be318638-e8a0-4c6d-977d-7a937aa84806>/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589>, 90bf5788-dca8-45e9-ab4f-7e4911f5e6f3 on vmstore-client-2 and 4fdb0da1-4cbe-4c1b-856c-b44158a1d276 on vmstore-client-1. [2019-04-29 09:35:19.946359] W [MSGID: 108027] [afr-common.c:2279:afr_attempt_readsubvol_set] 0-vmstore-replicate-0: no read subvols for (null) [2019-04-29 09:35:19.947943] E [MSGID: 108008] [afr-self-heal-common.c:352:afr_gfid_split_brain_source] 0-vmstore-replicate-0: Gfid mismatch detected for <gfid:be318638-e8a0-4c6d-977d-7a937aa84806>/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589>, 90bf5788-dca8-45e9-ab4f-7e4911f5e6f3 on vmstore-client-2 and 4fdb0da1-4cbe-4c1b-856c-b44158a1d276 on vmstore-client-0. [2019-04-29 09:35:19.948508] E [MSGID: 108008] [afr-self-heal-entry.c:260:afr_selfheal_detect_gfid_and_type_mismatch] 0-vmstore-replicate-0: Skipping conservative merge on the file. [2019-04-29 09:35:19.953275] E [MSGID: 108008] [afr-self-heal-common.c:352:afr_gfid_split_brain_source] 0-vmstore-replicate-0: Gfid mismatch detected for <gfid:be318638-e8a0-4c6d-977d-7a937aa84806>/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589>, 90bf5788-dca8-45e9-ab4f-7e4911f5e6f3 on vmstore-client-2 and 4fdb0da1-4cbe-4c1b-856c-b44158a1d276 on vmstore-client-1. [2019-04-29 09:35:19.954852] W [MSGID: 108027] [afr-common.c:2279:afr_attempt_readsubvol_set] 0-vmstore-replicate-0: no read subvols for (null) [2019-04-29 09:35:19.956447] E [MSGID: 108008] [afr-self-heal-common.c:352:afr_gfid_split_brain_source] 0-vmstore-replicate-0: Gfid mismatch detected for <gfid:be318638-e8a0-4c6d-977d-7a937aa84806>/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589>, 90bf5788-dca8-45e9-ab4f-7e4911f5e6f3 on vmstore-client-2 and 4fdb0da1-4cbe-4c1b-856c-b44158a1d276 on vmstore-client-0. [2019-04-29 09:35:19.956985] E [MSGID: 108008] [afr-self-heal-entry.c:260:afr_selfheal_detect_gfid_and_type_mismatch] 0-vmstore-replicate-0: Skipping conservative merge on the file. </snip>
Hi SAS, Can you provide the sos-reports and the getfattr outputs of the entry and its parent from all the nodes. Note: There are some corner cases which could lead to GFID mismatch. Some of them are listed in https://github.com/gluster/glusterfs/issues/502 and we are currently working on the design to fix these issues. Regards, Karthik
(In reply to Karthik U S from comment #4) > Hi SAS, > > Can you provide the sos-reports and the getfattr outputs of the entry and > its parent from all the nodes. > > Note: > There are some corner cases which could lead to GFID mismatch. Some of them > are listed in https://github.com/gluster/glusterfs/issues/502 and we are > currently working on the design to fix these issues. > > Regards, > Karthik I have provided the live system for debug to make the job easy, but for the case perspective, I will attach the required logs
Regarding the number of times, I could hit this issue, I was incorrect in comment0 I tried only once and hit the issue, and never tried to reproduce the issue. I can look for that information, once the initial investigation on the setup is complete
Hi Sas, In the fuse mount log I can see the following error messages: [2019-04-29 06:53:20.705277] W [MSGID: 108001] [afr-transaction.c:886:afr_handle_quorum] 0-vmstore-replicate-0: /.shard/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589: Failing MKNOD as quorum is not met [File exists] [2019-04-29 06:53:20.835201] E [MSGID: 108008] [afr-self-heal-common.c:352:afr_gfid_split_brain_source] 0-vmstore-replicate-0: Gfid mismatch detected for <gfid:be318638-e8a0-4c6d-977d-7a937aa84806>/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589>, 90bf5788-dca8-45e9-ab4f-7e4911f5e6f3 on vmstore-client-2 and 4fdb0da1-4cbe-4c1b-856c-b44158a1d276 on vmstore-client-1. [2019-04-29 06:53:20.837670] W [MSGID: 108027] [afr-common.c:2279:afr_attempt_readsubvol_set] 0-vmstore-replicate-0: no read subvols for /.shard/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589 From this it looks like, there were 2 mknods coming with the same file name i.e., "9bba57dc-f986-4dfe-a579-a5231b6bcbde.589" and first one succeeded on 2 of the bricks and the second one succeeded on the other one and failed on the 2 bricks with EEXIST, were the first mknod succeeded. After this the first GFID mismatch was reported. This looks like an orphaned entry. I do not see the original file for which this shard belongs to in any of the bricks and the shard size of this file is also 0. We are trying to handle this kind of scenarios as well in the issue mentioned in comment #4. Can you try to reproduce this issue and let us know the frequency of occurrence? Regards, Karthik
(In reply to Karthik U S from comment #7) > Hi Sas, > > In the fuse mount log I can see the following error messages: > > [2019-04-29 06:53:20.705277] W [MSGID: 108001] > [afr-transaction.c:886:afr_handle_quorum] 0-vmstore-replicate-0: > /.shard/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589: Failing MKNOD as quorum is > not met [File exists] > [2019-04-29 06:53:20.835201] E [MSGID: 108008] > [afr-self-heal-common.c:352:afr_gfid_split_brain_source] > 0-vmstore-replicate-0: Gfid mismatch detected for > <gfid:be318638-e8a0-4c6d-977d-7a937aa84806>/9bba57dc-f986-4dfe-a579- > a5231b6bcbde.589>, 90bf5788-dca8-45e9-ab4f-7e4911f5e6f3 on vmstore-client-2 > and 4fdb0da1-4cbe-4c1b-856c-b44158a1d276 on vmstore-client-1. > [2019-04-29 06:53:20.837670] W [MSGID: 108027] > [afr-common.c:2279:afr_attempt_readsubvol_set] 0-vmstore-replicate-0: no > read subvols for /.shard/9bba57dc-f986-4dfe-a579-a5231b6bcbde.589 > > From this it looks like, there were 2 mknods coming with the same file name > i.e., "9bba57dc-f986-4dfe-a579-a5231b6bcbde.589" and first one succeeded on > 2 of the bricks and the second one succeeded on the other one and failed on > the 2 bricks with EEXIST, were the first mknod succeeded. After this the > first GFID mismatch was reported. > > This looks like an orphaned entry. I do not see the original file for which > this shard belongs to in any of the bricks and the shard size of this file > is also 0. > > We are trying to handle this kind of scenarios as well in the issue > mentioned in comment #4. Can you try to reproduce this issue and let us know > the frequency of occurrence? > > Regards, > Karthik Thanks Karthik, I can look in to reproducing this issue with the RHHI-V setup.