Bug 1159205

Summary: glusterd/geo-rep: Few files are not synced to slave when files are being created during geo-rep start
Product: [Community] GlusterFS Reporter: Kotresh HR <khiremat>
Component: glusterdAssignee: Kotresh HR <khiremat>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.0CC: avishwan, bugs, chrisw, csaba, khiremat, mzywusko, nlevinki, nsathyan, rabhat, vbhat
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-v3.6.2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1139196 Environment:
Last Closed: 2016-02-04 15:05:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1139156, 1139196    
Bug Blocks:    

Description Kotresh HR 2014-10-31 08:36:56 UTC
+++ This bug was initially created as a clone of Bug #1139196 +++

+++ This bug was initially created as a clone of Bug #1139156 +++

Description of problem:
If the I/O is running during the geo-rep start time, then few of the files are not synced to slave.


Version-Release number of selected component (if applicable):
glusterfs-3.6.0.28-1.el6rhs.x86_64

How reproducible:
Seems to be reproducible frequently.

Steps to Reproduce:
1. create master and slave volume and start file creation in master.
2. Create and start geo-rep.
3. Wait for files to sync to slave.

Actual results:
Few files are skipped while syncing. As a result, slave will not be having all the files in master.

Expected results:
All the files in master should be synced to slave in any case.

Additional info:


log info from one of the log.

[2014-09-08 12:51:40.558691] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/9611d805-4647-4647-b969-0f5596f8591c', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x0046a7f526-fbac-41f8-ac18-2c46b05bfddc\x00\x00\x00\x81\x80tc-actions-env-rules.txt\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...
[2014-09-08 12:51:45.628182] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/9611d805-4647-4647-b969-0f5596f8591c', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x004abd9cb1-a7dd-4228-aa47-873c343ebcfd\x00\x00\x00A\xc0timestamping\x00\x00\x00\x01\xc0\x00\x00\x00\x00'])...
[2014-09-08 12:51:50.994118] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/4abd9cb1-a7dd-4228-aa47-873c343ebcfd', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x0037d26f46-d8d3-4cec-90b1-87c5efd45eb3\x00\x00\x00\x81\x80.gitignore\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...
[2014-09-08 12:51:56.281146] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/4abd9cb1-a7dd-4228-aa47-873c343ebcfd', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x000a373591-3499-4642-94b3-7858227a9de0\x00\x00\x00\x81\x80timestamping.c\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...
[2014-09-08 12:52:01.956902] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/9611d805-4647-4647-b969-0f5596f8591c', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x0064cecf23-d45e-4976-aa99-c75eb23c2f90\x00\x00\x00\x81\x80tlan.txt\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...
[2014-09-08 12:52:07.236882] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/9611d805-4647-4647-b969-0f5596f8591c', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x002c7f5d32-ece4-4288-bf00-aeb79316138b\x00\x00\x00\x81\x80vxge.txt\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...

Comment 1 Anand Avati 2014-10-31 08:41:41 UTC
REVIEW: http://review.gluster.org/9023 (geo-rep/glusterd: Enable changelog and marker during geo-rep create.) posted (#1) for review on release-3.6 by Kotresh HR (khiremat)

Comment 2 Anand Avati 2014-11-13 01:57:04 UTC
COMMIT: http://review.gluster.org/9023 committed in release-3.6 by Venky Shankar (vshankar) 
------
commit 97f021f37d388f903798d78fbda471efb3fb92c0
Author: Kotresh HR <khiremat>
Date:   Mon Sep 8 15:04:15 2014 +0530

    geo-rep/glusterd: Enable changelog and marker during geo-rep create.
    
    PROBLEM:
    Geo-rep misses few a files to sync when I/O happenned during
    geo-rep start.
    
    ANALYSES:
    To use the available changelogs to handle deletes/renames,
    'xsync upper limit' is introduced which limits the xsync
    crawl till the changelog register time.  But there is a
    small time interval between the changelog register time and
    the time changelog actually enabled. If there is I/O between
    this interval, it will not be synced through xsync as it is
    beyond changelog register time and not through changelog also
    as changelog is not actually enabled.
    
    SOLUTION:
    Enable changelog and marker during geo-rep create instead
    of geo-rep start so that entries are captured in changelog
    and above said interval is nullified.
    
    BUG: 1159205
    Change-Id: If5203eb1cfcbde3999f97a5f1a6a1af4875ac358
    Reviewed-on: http://review.gluster.org/8650
    Signed-off-by: Kotresh HR <khiremat>
    Reviewed-on: http://review.gluster.org/9023
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Venky Shankar <vshankar>
    Tested-by: Venky Shankar <vshankar>

Comment 3 Kaushal 2016-02-04 15:05:34 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v3.6.2, please open a new bug report.

glusterfs-v3.6.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/5978top
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 4 Kaushal 2016-02-04 15:08:06 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v3.6.2, please open a new bug report.

glusterfs-v3.6.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2015/01/glusterfs-3-6-2-ga-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user