Bug 1200733 - [geo-rep]: some of the pre existing files before geo-rep session did not sync to slave
Summary: [geo-rep]: some of the pre existing files before geo-rep session did not sync...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: geo-replication
Version: mainline
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Aravinda VK
QA Contact:
URL:
Whiteboard:
Depends On: 1198056
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-11 10:01 UTC by Aravinda VK
Modified: 2015-05-14 17:35 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.7.0beta1
Doc Type: Bug Fix
Doc Text:
Clone Of: 1198056
Environment:
Last Closed: 2015-05-14 17:26:52 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Aravinda VK 2015-03-11 10:01:40 UTC
+++ This bug was initially created as a clone of Bug #1198056 +++

Description of problem:
=======================

Some of the files which were present before the geo-rep session was created. The files didn't sync to slave until setfattr was performed on those files.


Master volume had the following 4 files:
=========================================

[root@wingo master]# ls
host.conf  hosts  hosts.allow  hosts.deny
[root@wingo master]#

Steps carried
=============

1. Create 4 node cluster (master cluster)
2. Create a 2x2 volume (gr)
3. Start the volume
4. Mount the volume on client (/mnt/master)
5. cp -rf /etc/host* /mnt/master/
6. Create another 4 node cluster (slave cluster)
7. Create a 1x2 volume (gr) from node 1 and node 2 of slave cluster
8. Start the volume
9. Mount the volume on client (/mnt/slave)
10. Create password less ssh between node 1 of master and node 2 of slave.
11. Execute gluster system:: execute gsec_create on node 1 of master
12. Create a geo-rep session between node 1 of master to node 1 of slave.
gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> create push-pem [force]
13. Start a geo-rep session 
gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> start
14. ls to the slave volume (ls /mnt/slave)

Actual results:
===============

2 files "host.conf and hosts" were synced to slave and 2 files "hosts.allow and hosts.deny" didn't sync to slave volume.

Performed "setfattr -n glusterfs.geo-rep.trigger-sync -v "1" /mnt/master/hosts.deny" and 
"setfattr -n glusterfs.geo-rep.trigger-sync -v "1" /mnt/master/hosts.allow"

after which the files were synced to /mnt/slave


Expected results:
=================

All the files should be successfully synced to slave volume.

Version-Release number of selected component (if applicable):
=============================================================


How reproducible:
=================

Tried once.

Comment 1 Anand Avati 2015-03-11 10:20:08 UTC
REVIEW: http://review.gluster.org/9855 (geo-rep: Do not use xsync_upper_limit for change detection) posted (#1) for review on master by Aravinda VK (avishwan@redhat.com)

Comment 2 Aravinda VK 2015-03-11 10:21:27 UTC
Root caused the issue:

If files created before Geo-rep create, xtime xattr will not be available. After geo-rep create it enables indexing. Marker will update xtime for all the files which are created/modified after indexing is enabled.

Geo-rep uses changelog register time as upper limit for xsync to decide a file should be synced or not. It compares xtime of each file >= upper_limit, If xtime is not found it updates xtime with current time. But this updated xtime is greater than upper_limit, xsync skips this file by thinking that changelog will pick this. But changelog will not have CREATE info for this file only SETXATTR/DATA is recorded.

Patch sent to fix this issue: 
http://review.gluster.org/#/c/9855/

Comment 3 Anand Avati 2015-03-12 11:04:38 UTC
REVIEW: http://review.gluster.org/9855 (geo-rep: Do not use xsync_upper_limit for change detection) posted (#2) for review on master by Aravinda VK (avishwan@redhat.com)

Comment 4 Anand Avati 2015-03-16 05:55:36 UTC
COMMIT: http://review.gluster.org/9855 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 7d8be3613f7384f5118f26e194fe7c64ea69d11c
Author: Aravinda VK <avishwan@redhat.com>
Date:   Wed Mar 11 13:31:09 2015 +0530

    geo-rep: Do not use xsync_upper_limit for change detection
    
    Use register time(xsync_upper_limit) only for stime update, do not
    use for change detection.
    
    Problem 1:
    If a file created before geo-rep, xtime xattr does not exist.
    Geo-rep updates xtime of the file to current time if not exists.
    xtime > upper_limit so geo-rep will not pick those files. Changelog
    either will have SETXATTR, and fails to sync the file.
    
    Problem 2:
    If a file is created before geo-rep create and updated after
    geo-rep start. xtime of the file is greater than upper limit(geo-rep
    start time/changelog register time). Geo-rep(XSync) will not pick this
    file for syncing. Changelog will have only DATA recorded for that file.
    Geo-rep tries DATA without any ENTRY ops and fails with rsync error.
    
    BUG: 1200733
    Change-Id: Ie4e8f284db689d2c755ef8e7ecbb658db1c0785f
    Signed-off-by: Aravinda VK <avishwan@redhat.com>
    Reviewed-on: http://review.gluster.org/9855
    Reviewed-by: Kotresh HR <khiremat@redhat.com>
    Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com>
    Tested-by: Saravanakumar Arumugam <sarumuga@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Comment 5 Niels de Vos 2015-05-14 17:26:52 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 6 Niels de Vos 2015-05-14 17:28:28 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 7 Niels de Vos 2015-05-14 17:35:17 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.