Bug 1181669

Description of problem:

I have a replica 2 setup as follows:
------------
root@gfs_serv0:~] gluster v info gfs_replicated_vol
 
Volume Name: gfs_replicated_vol
Type: Replicate
Volume ID: 4e72e0cc-318f-4706-92af-4a56fc793063
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gfs_serv0:/mnt/bricks/gfs_replicated_vol/brick
Brick2: gfs_serv1:/mnt/bricks/gfs_replicated_vol/brick
Options Reconfigured:
nfs.disable: on
cluster.self-heal-daemon: on
nfs.enable-ino32: on
network.ping-timeout: 10
root@gfs_serv0:~]

I check for heal info and I find that all files are in sync:
------------
root@gfs_serv0:~] gluster volume heal gfs_replicated_vol info


Gathering Heal info on volume _tftpboot has been successful

Brick gfs_serv0:/mnt/bricks/gfs_replicated_vol/brick

Number of entries: 0


Brick gfs_serv1:/mnt/bricks/gfs_replicated_vol/brick

Number of entries: 0

root@gfs_serv0:~] 

However, for a few files, I see that the md5sum of their replicas (in the two bricks) differ (even as their sizes are identical). To illustrate,
------------
root@gfs_serv0:~] ls -li /mnt/bricks/gfs_replicated_vol/brick/bin/bash

4187497 -rwxr-xr-x 2 root root 666648 Dec  6 22:48 /mnt/bricks/gfs_replicated_vol/brick/bin/bash

root@gfs_serv0:~] md5sum /mnt/bricks/gfs_replicated_vol/brick/bin/bash

fc61db7be6eeda79f0b0bff58e622ace  /mnt/bricks/gfs_replicated_vol/brick/bin/bash

root@gfs_serv1:~] ls -li /mnt/bricks/gfs_replicated_vol/brick/bin/bash

8389144 -rwxr-xr-x 2 root root 666648 Dec  6 22:48 /mnt/bricks/gfs_replicated_vol/brick/bin/bash

root@gfs_serv1:~] md5sum /mnt/bricks/gfs_replicated_vol/brick/bin/bash

154b9852621b1651aff4af0764897c9a  /mnt/bricks/gfs_replicated_vol/brick/bin/bash

Further information on the two replicas:
------------
root@gfs_serv0:~] getfattr -m . -d -e hex /mnt/bricks/gfs_replicated_vol/brick/bin/bash

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/gfs_replicated_vol/brick/bin/bash

trusted.afr.gfs_replicated_vol-client-0=0x000000000000000000000000

trusted.afr.gfs_replicated_vol-client-1=0x000000000000000000000000

trusted.gfid=0xc577569a74cb4f23825daef95e9dcbb4



root@gfs_serv1:~] getfattr -m . -d -e hex /mnt/bricks/gfs_replicated_vol/brick/bin/bash

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/gfs_replicated_vol/brick/bin/bash

trusted.afr.gfs_replicated_vol-client-0=0x000000000000000000000000

trusted.afr.gfs_replicated_vol-client-1=0x000000000000000000000000

trusted.gfid=0xc577569a74cb4f23825daef95e9dcbb4


I tried to force a heal by triggering a lookup on the file using ls -l /gfs_replicated_vol/brick/bin/bash but that made no difference.

Version-Release number of selected component (if applicable):


How reproducible: Intermittent. 


Steps to Reproduce:

I do not have a simple way to replicate this - just something we see on our servers on rare days.

Actual results:

Files are not in-sync even as gluster volume heal info does not report a single entry.

Expected results:

In the case that files are not in sync, glusterfs should be able to identify them and report them to the system administrator under the heal info command.

Additional info:

None