Description of problem:

Feature request really.  As per IRC discussion:

<jkroon> a useful adition would be to be able to query WHY it things the file is is split brain, eg: volume heal ${volname} info file <gfid:xxxxx>

Basically, given a path or gfid that's in split-brain, a way to dump the information required to make an informed decision on what to do would be extremely helpful.

A slightly improved way to resolve a gfid split (as what I've seen last night) where a file path has different gfid values on different bricks would be most welcome (the way I used - described lower down - is likely to be very error prone unless you're particularly careful).

This follows a rather long discussion regarding split brains a few hours ago.  full discussion below.

<jkroon> JoeJulian, whilst i've got you - i've read your blog on the whole gfid thing a few times now, and from the output of volume heal ${volname} info split-brain I'm still a little baffled as to how to get those resolved gfid ones sorted out in a sensible manner.
<jkroon> mostly what I do is I run a find on one of the underlying bricks with (assuming pwd is the brick folder) find . -samefile .gluster/aa/bb/fullgfid ... which then gives me the path.
<jkroon> I think that assumes it's a file and not a folder ?
<jkroon> but with slightly over 5m files on the brick that can take a VERY long time.
<jkroon> I'm guessing I can just compare meta-data on the gfid file itself already ... but in case where I randomly shoot one of the two files (in the cases I've had it's mostly log files so both are really wrong)
<JoeJulian> except folders are symlinks in the .glusterfs tree so you can't really check the meta-data.
<jkroon> but also - in order to rm one of the two, you need all hard links to it, so my current concede rm's both the gfid file and the other file - last time I needed that was with 3.5
<jkroon> symlinks to the actual folders?
<jkroon> doesn't that mean we can use readlink -f to get to the real path?
<JoeJulian> There's really no valid reason for folders to go split-brain. iirc, there's an open bug about them (unless it was fixed recently)
<jkroon> folders  that goes into split brain I've only ever seen for two reasons:  1.  permissions differ on the bricks; 2.  time-stamps differ.
<JoeJulian> I've seen it for no reason at all.
* ChanServ gives channel operator status to hagarth
<JoeJulian> Just there was a trusted.afr attribute on both replica stating the other one was out of date, but they completely matched.
<jkroon> theoretically ANY meta data differences can cause it.  i also read something about files being modified could trigger it - but newer versions handles that specific cse.
<jkroon> i've got 19 splits at the moment on one cluster, those that are not gfids are all folders.
<JoeJulian> I fix folder splits by removing all trusted.afr from all directories on the bricks.
<JoeJulian> @split-brain
<glusterbot> JoeJulian: (#1) To heal split brain, rtfm at http://gluster.readthedocs.io/en/latest/Troubleshooting/heal-info-and-split-brain-resolution/ first., or (#2) For versions older than 3.7, see splitmount https://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/, or (#3) For manual brick-level recovery see http://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/
<jkroon> JoeJulian, that first link actually perhaps explains what you're seeing.
<JoeJulian> Of course even the manual repair documentation is wrong for directories.
<jkroon> the folder goes into split-brain due to file in folder being in gfid split brain.
<JoeJulian> Not in any of the cases I've looked at.
<JoeJulian> I have seen gfid splits, but that usually looks really ugly, multiple copies of filenames and stuff.
<jkroon> a useful adition would be to be able to query WHY it things the file is is split brain, eg: volume heal ${volname} info file <gfid:xxxxx>
<JoeJulian> jkroon: good idea, file a bug report
<glusterbot> https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
<jkroon> JoeJulian, if the "volume ${volname} split-brain bigger-file /path/to/file" where file is not listed as being in split brain, but is the cause of the folder split brain refuses to heal - i'm guessing it's manual repair time by removing it straight from the brick?
<jkroon> is it possible / why would the same file path map to two different gfid values on different bricks?
* plarsen (~plarsen@redhat/jboss/pdpc.professional.plarsen) has joined #gluster
<JoeJulian> I haven't had a chance to try the setfattr method of repairing split-brain on a directory yet.
<JoeJulian> Brick A is up (no quorum) and you create directory "foo". Brick "A" goes down and brick "B" comes up. "foo" doesn't exist so you create it. Now it's been created on two different bricks and each has been assigned a new gfid.
<jkroon> oh boy.
<jkroon> ok, in my case now it's a file, but yes, that's a very viable scenario seeing that I had a junior tech that wreacked havoc on that system in particular.
<jkroon> yea, they're all a bunch of courierpop3dsizelist files inside folders that's causing problems.
<jkroon> can't heal them because they have different gfid values.
<jkroon> and can't rm them via the mountpoint.

After this I manually tracked the problem-causing files using readlink -f .gluster/.../${gfid} as reported by volume heal ... info split-brain and then using ls on the mountpoint to find the files causing IO errors, and then on each brick used getfattr to get the two different gfid values, then I removed (due to not caring about the contents of the files - they'd be recreated in all cases) both the gfid file as well as it's associated file from the bricks.  After this sanity returned.

Point being, a way to determine WHAT is the reason for a file being reported as split-brain would be extremely useful.  In the above scenario it could simply state something like "file xyz in /path has differing gfid values: on server1:/brick/path=gfid, server2:/brick/path=gfid" type of thing, or if the files has differering content, provide the mtimes and file sizes.  This should make it much easier and faster to decide what to do, not to mention providing a little more detail.  In the case of gfid values being supplied it's probably a good idea if we can (within reasonable time frames) resolve that gfid to an actual path.