Bug 1001502

Summary: Split brain not detected in a replica 3 volume
Product: [Community] GlusterFS Reporter: Joe Julian <joe>
Component: replicateAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED NOTABUG QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: gluster-bugs, ravishankar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1001540 (view as bug list) Environment:
Last Closed: 2013-10-16 09:06:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1001540    

Description Joe Julian 2013-08-27 07:39:59 UTC 
Description of problem:
One file in a replica 3 volume is marked dirty for all three bricks (on all three bricks)

# file: /data/glusterfs/vmimages/a/brick/mysql1.img
trusted.afr.vmimages-client-0=0x000000050000000000000000
trusted.afr.vmimages-client-1=0x000000010000000000000000
trusted.afr.vmimages-client-2=0x000000010000000000000000

The file shows in heal info:

Brick ewcs2:/data/glusterfs/vmimages/a/brick
Number of entries: 1
/mysql1.img

Brick ewcs10:/data/glusterfs/vmimages/a/brick
Number of entries: 1
/mysql1.img

Brick ewcs7:/data/glusterfs/vmimages/a/brick
Number of entries: 1
/mysql1.img

After neither a heal nor a heal full does volume heal info split-brain list the file.

Version-Release number of selected component (if applicable):
3.4.0

How reproducible:
Always

Steps to Reproduce:
1.Create a replica 3 volume with a file on it
2.stop the volume
3.mark all three xattrs on all three replicas
4.start the volume
5.gluster volume heal $vol full

Actual results:
No split-brain detected

Expected results:
Split brain should be detected.

Additional info:
Comment 1 Ravishankar N 2013-10-11 12:22:42 UTC 
Hi Joe,
The scenario of non-zeroes for all clients in the xattr of all bricks is considered a fool-fool scenario (and not a split-brain). i.e. for a 2 way AFR, the following null pending matrix is considered fool-fool.
       [ [1,15] [1,1] ]
where brick1 has xattrs [1,15] and brick 2 has [1,1].
In such cases AFR just chooses the brick whose change log indicates maximum pending transactions for other bricks as the source and starts the heal.In the above matrix, brick1 is chosen as source.

I tried the test procedure you described on glusterfs mainline but was unable to hit the issue. The heal command cleared the xattrs of all bricks.
From glustershd.log:
[2013-10-11 11:19:31.270197] I [afr-self-heal-common.c:2811:afr_log_self_heal_completion_status] 0-repvol-replicate-0:  foreground data self heal  is successfully completed,  data self heal from repvol-client-2  to sinks  repvol-client-0, repvol-client-1, with 5 bytes on repvol-client-0, 5 bytes on repvol-client-1, 5 bytes on repvol-client-2,  data - Pending matrix:  [ [ 80 16 16 ] [ 80 16 16 ] [ 80 16 16 ] ]  on /file