Bug 1139196

Summary:	dist-geo-rep: Few files are not synced to slave when files are being created during geo-rep start
Product:	[Community] GlusterFS	Reporter:	Kotresh HR <khiremat>
Component:	glusterd	Assignee:	Kotresh HR <khiremat>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	aavati, avishwan, csaba, gluster-bugs, khiremat, nlevinki, nsathyan, ssamanta, vbhat
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1139156
Clones:	1159205 (view as bug list)		Environment:
Last Closed:	2015-05-14 17:27:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1139156
Bug Blocks:	1159205

Description Kotresh HR 2014-09-08 11:10:55 UTC

+++ This bug was initially created as a clone of Bug #1139156 +++

Description of problem:
If the I/O is running during the geo-rep start time, then few of the files are not synced to slave.


Version-Release number of selected component (if applicable):
glusterfs-3.6.0.28-1.el6rhs.x86_64

How reproducible:
Seems to be reproducible frequently.

Steps to Reproduce:
1. create master and slave volume and start file creation in master.
2. Create and start geo-rep.
3. Wait for files to sync to slave.

Actual results:
Few files are skipped while syncing. As a result, slave will not be having all the files in master.

Expected results:
All the files in master should be synced to slave in any case.

Additional info:


log info from one of the log.

[2014-09-08 12:51:40.558691] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/9611d805-4647-4647-b969-0f5596f8591c', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x0046a7f526-fbac-41f8-ac18-2c46b05bfddc\x00\x00\x00\x81\x80tc-actions-env-rules.txt\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...
[2014-09-08 12:51:45.628182] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/9611d805-4647-4647-b969-0f5596f8591c', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x004abd9cb1-a7dd-4228-aa47-873c343ebcfd\x00\x00\x00A\xc0timestamping\x00\x00\x00\x01\xc0\x00\x00\x00\x00'])...
[2014-09-08 12:51:50.994118] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/4abd9cb1-a7dd-4228-aa47-873c343ebcfd', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x0037d26f46-d8d3-4cec-90b1-87c5efd45eb3\x00\x00\x00\x81\x80.gitignore\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...
[2014-09-08 12:51:56.281146] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/4abd9cb1-a7dd-4228-aa47-873c343ebcfd', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x000a373591-3499-4642-94b3-7858227a9de0\x00\x00\x00\x81\x80timestamping.c\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...
[2014-09-08 12:52:01.956902] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/9611d805-4647-4647-b969-0f5596f8591c', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x0064cecf23-d45e-4976-aa99-c75eb23c2f90\x00\x00\x00\x81\x80tlan.txt\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...
[2014-09-08 12:52:07.236882] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/9611d805-4647-4647-b969-0f5596f8591c', 'glusterfs.gfid.newfile', '\x00\x00\x00\x00\x00\x00\x00\x002c7f5d32-ece4-4288-bf00-aeb79316138b\x00\x00\x00\x81\x80vxge.txt\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x00\x00'])...

Comment 1 Anand Avati 2014-09-08 11:15:41 UTC

REVIEW: http://review.gluster.org/8650 (geo-rep/glusterd: Enable changelog and marker during geo-rep create.) posted (#1) for review on master by Kotresh HR (khiremat)

Comment 2 Anand Avati 2014-10-27 13:41:13 UTC

COMMIT: http://review.gluster.org/8650 committed in master by Venky Shankar (vshankar) 
------
commit 5c6e88320489d789fcd026bed72009b0806fe314
Author: Kotresh HR <khiremat>
Date:   Mon Sep 8 15:04:15 2014 +0530

    geo-rep/glusterd: Enable changelog and marker during geo-rep create.
    
    PROBLEM:
    Geo-rep misses few a files to sync when I/O happenned during
    geo-rep start.
    
    ANALYSES:
    To use the available changelogs to handle deletes/renames,
    'xsync upper limit' is introduced which limits the xsync
    crawl till the changelog register time.  But there is a
    small time interval between the changelog register time and
    the time changelog actually enabled. If there is I/O between
    this interval, it will not be synced through xsync as it is
    beyond changelog register time and not through changelog also
    as changelog is not actually enabled.
    
    SOLUTION:
    Enable changelog and marker during geo-rep create instead
    of geo-rep start so that entries are captured in changelog
    and above said interval is nullified.
    
    Change-Id: Ic5f0457a4b67a335cbbb37d34db5f8cb8bc901c4
    BUG: 1139196
    Signed-off-by: Kotresh HR <khiremat>
    Reviewed-on: http://review.gluster.org/8650
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Aravinda VK <avishwan>
    Reviewed-by: Avra Sengupta <asengupt>
    Reviewed-by: Venky Shankar <vshankar>
    Tested-by: Venky Shankar <vshankar>

Comment 3 Niels de Vos 2015-05-14 17:27:31 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 4 Niels de Vos 2015-05-14 17:35:35 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 5 Niels de Vos 2015-05-14 17:37:57 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 6 Niels de Vos 2015-05-14 17:43:41 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user