Bug 1200733

Summary:	[geo-rep]: some of the pre existing files before geo-rep session did not sync to slave
Product:	[Community] GlusterFS	Reporter:	Aravinda VK <avishwan>
Component:	geo-replication	Assignee:	Aravinda VK <avishwan>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	aavati, bugs, csaba, gluster-bugs, nlevinki, rhinduja, rhs-bugs, sanandpa, smanjara, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.0beta1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1198056	Environment:
Last Closed:	2015-05-14 17:26:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1198056
Bug Blocks:

Description Aravinda VK 2015-03-11 10:01:40 UTC

+++ This bug was initially created as a clone of Bug #1198056 +++

Description of problem:
=======================

Some of the files which were present before the geo-rep session was created. The files didn't sync to slave until setfattr was performed on those files.


Master volume had the following 4 files:
=========================================

[root@wingo master]# ls
host.conf  hosts  hosts.allow  hosts.deny
[root@wingo master]#

Steps carried
=============

1. Create 4 node cluster (master cluster)
2. Create a 2x2 volume (gr)
3. Start the volume
4. Mount the volume on client (/mnt/master)
5. cp -rf /etc/host* /mnt/master/
6. Create another 4 node cluster (slave cluster)
7. Create a 1x2 volume (gr) from node 1 and node 2 of slave cluster
8. Start the volume
9. Mount the volume on client (/mnt/slave)
10. Create password less ssh between node 1 of master and node 2 of slave.
11. Execute gluster system:: execute gsec_create on node 1 of master
12. Create a geo-rep session between node 1 of master to node 1 of slave.
gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> create push-pem [force]
13. Start a geo-rep session 
gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> start
14. ls to the slave volume (ls /mnt/slave)

Actual results:
===============

2 files "host.conf and hosts" were synced to slave and 2 files "hosts.allow and hosts.deny" didn't sync to slave volume.

Performed "setfattr -n glusterfs.geo-rep.trigger-sync -v "1" /mnt/master/hosts.deny" and 
"setfattr -n glusterfs.geo-rep.trigger-sync -v "1" /mnt/master/hosts.allow"

after which the files were synced to /mnt/slave


Expected results:
=================

All the files should be successfully synced to slave volume.

Version-Release number of selected component (if applicable):
=============================================================


How reproducible:
=================

Tried once.

Comment 1 Anand Avati 2015-03-11 10:20:08 UTC

REVIEW: http://review.gluster.org/9855 (geo-rep: Do not use xsync_upper_limit for change detection) posted (#1) for review on master by Aravinda VK (avishwan)

Comment 2 Aravinda VK 2015-03-11 10:21:27 UTC

Root caused the issue:

If files created before Geo-rep create, xtime xattr will not be available. After geo-rep create it enables indexing. Marker will update xtime for all the files which are created/modified after indexing is enabled.

Geo-rep uses changelog register time as upper limit for xsync to decide a file should be synced or not. It compares xtime of each file >= upper_limit, If xtime is not found it updates xtime with current time. But this updated xtime is greater than upper_limit, xsync skips this file by thinking that changelog will pick this. But changelog will not have CREATE info for this file only SETXATTR/DATA is recorded.

Patch sent to fix this issue: 
http://review.gluster.org/#/c/9855/

Comment 3 Anand Avati 2015-03-12 11:04:38 UTC

REVIEW: http://review.gluster.org/9855 (geo-rep: Do not use xsync_upper_limit for change detection) posted (#2) for review on master by Aravinda VK (avishwan)

Comment 4 Anand Avati 2015-03-16 05:55:36 UTC

COMMIT: http://review.gluster.org/9855 committed in master by Vijay Bellur (vbellur) 
------
commit 7d8be3613f7384f5118f26e194fe7c64ea69d11c
Author: Aravinda VK <avishwan>
Date:   Wed Mar 11 13:31:09 2015 +0530

    geo-rep: Do not use xsync_upper_limit for change detection
    
    Use register time(xsync_upper_limit) only for stime update, do not
    use for change detection.
    
    Problem 1:
    If a file created before geo-rep, xtime xattr does not exist.
    Geo-rep updates xtime of the file to current time if not exists.
    xtime > upper_limit so geo-rep will not pick those files. Changelog
    either will have SETXATTR, and fails to sync the file.
    
    Problem 2:
    If a file is created before geo-rep create and updated after
    geo-rep start. xtime of the file is greater than upper limit(geo-rep
    start time/changelog register time). Geo-rep(XSync) will not pick this
    file for syncing. Changelog will have only DATA recorded for that file.
    Geo-rep tries DATA without any ENTRY ops and fails with rsync error.
    
    BUG: 1200733
    Change-Id: Ie4e8f284db689d2c755ef8e7ecbb658db1c0785f
    Signed-off-by: Aravinda VK <avishwan>
    Reviewed-on: http://review.gluster.org/9855
    Reviewed-by: Kotresh HR <khiremat>
    Reviewed-by: Saravanakumar Arumugam <sarumuga>
    Tested-by: Saravanakumar Arumugam <sarumuga>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 5 Niels de Vos 2015-05-14 17:26:52 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 6 Niels de Vos 2015-05-14 17:28:28 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 7 Niels de Vos 2015-05-14 17:35:17 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user