Bug 1001502 - Split brain not detected in a replica 3 volume
Split brain not detected in a replica 3 volume
Product: GlusterFS
Classification: Community
Component: replicate (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Pranith Kumar K
Depends On:
Blocks: 1001540
  Show dependency treegraph
Reported: 2013-08-27 03:39 EDT by Joe Julian
Modified: 2013-10-16 05:06 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1001540 (view as bug list)
Last Closed: 2013-10-16 05:06:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Joe Julian 2013-08-27 03:39:59 EDT
Description of problem:
One file in a replica 3 volume is marked dirty for all three bricks (on all three bricks)

# file: /data/glusterfs/vmimages/a/brick/mysql1.img

The file shows in heal info:

Brick ewcs2:/data/glusterfs/vmimages/a/brick
Number of entries: 1

Brick ewcs10:/data/glusterfs/vmimages/a/brick
Number of entries: 1

Brick ewcs7:/data/glusterfs/vmimages/a/brick
Number of entries: 1

After neither a heal nor a heal full does volume heal info split-brain list the file.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.Create a replica 3 volume with a file on it
2.stop the volume
3.mark all three xattrs on all three replicas
4.start the volume
5.gluster volume heal $vol full

Actual results:
No split-brain detected

Expected results:
Split brain should be detected.

Additional info:
Comment 1 Ravishankar N 2013-10-11 08:22:42 EDT
Hi Joe,
The scenario of non-zeroes for all clients in the xattr of all bricks is considered a fool-fool scenario (and not a split-brain). i.e. for a 2 way AFR, the following null pending matrix is considered fool-fool.
       [ [1,15] [1,1] ]
where brick1 has xattrs [1,15] and brick 2 has [1,1].
In such cases AFR just chooses the brick whose change log indicates maximum pending transactions for other bricks as the source and starts the heal.In the above matrix, brick1 is chosen as source.

I tried the test procedure you described on glusterfs mainline but was unable to hit the issue. The heal command cleared the xattrs of all bricks.
From glustershd.log:
[2013-10-11 11:19:31.270197] I [afr-self-heal-common.c:2811:afr_log_self_heal_completion_status] 0-repvol-replicate-0:  foreground data self heal  is successfully completed,  data self heal from repvol-client-2  to sinks  repvol-client-0, repvol-client-1, with 5 bytes on repvol-client-0, 5 bytes on repvol-client-1, 5 bytes on repvol-client-2,  data - Pending matrix:  [ [ 80 16 16 ] [ 80 16 16 ] [ 80 16 16 ] ]  on /file

Note You need to log in before you can comment on or make changes to this bug.