Bug 1187140

Summary: [RFE]: geo-rep: Tool to find missing files in slave volume
Product: [Community] GlusterFS Reporter: Kotresh HR <khiremat>
Component: geo-replicationAssignee: Kotresh HR <khiremat>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: mainlineCC: avishwan, bugs, gluster-bugs, ndevos
Target Milestone: ---Keywords: FutureFeature, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.0beta1 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-05-14 17:26:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kotresh HR 2015-01-29 12:09:32 UTC
Description of problem:

Need a tool to find missing files in slave volume.
Now if geo-rep misses a file to sync, whole of gfids from changelog
is dumped. There is no way we can know the exact files that missed
to sync.

Comment 1 Anand Avati 2015-01-29 12:10:11 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#1) for review on master by Kotresh HR (khiremat)

Comment 2 Anand Avati 2015-02-04 12:01:10 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#2) for review on master by Kotresh HR (khiremat)

Comment 3 Anand Avati 2015-02-05 08:01:05 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#3) for review on master by Aravinda VK (avishwan)

Comment 4 Anand Avati 2015-02-05 09:38:27 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#4) for review on master by Aravinda VK (avishwan)

Comment 5 Anand Avati 2015-02-05 13:24:05 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#5) for review on master by Aravinda VK (avishwan)

Comment 6 Anand Avati 2015-02-05 13:53:34 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#6) for review on master by Aravinda VK (avishwan)

Comment 7 Anand Avati 2015-03-06 03:08:02 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#7) for review on master by Aravinda VK (avishwan)

Comment 8 Anand Avati 2015-03-11 10:12:18 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#8) for review on master by Kotresh HR (khiremat)

Comment 9 Anand Avati 2015-03-12 10:57:45 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#9) for review on master by Kotresh HR (khiremat)

Comment 10 Anand Avati 2015-03-13 04:10:10 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#10) for review on master by Kotresh HR (khiremat)

Comment 11 Anand Avati 2015-03-13 05:35:11 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#11) for review on master by Kotresh HR (khiremat)

Comment 12 Anand Avati 2015-03-13 05:48:18 UTC
REVIEW: http://review.gluster.org/9503 (tools: Finds missing files in gluster volume given backend brickpath) posted (#12) for review on master by Kotresh HR (khiremat)

Comment 13 Anand Avati 2015-03-16 04:20:23 UTC
COMMIT: http://review.gluster.org/9503 committed in master by Vijay Bellur (vbellur) 
------
commit 7a9a66cc5fb7f06118fab1fc2ae1c43cfbb1178f
Author: Kotresh HR <khiremat>
Date:   Thu Jan 29 15:53:19 2015 +0530

    tools: Finds missing files in gluster volume given backend brickpath
    
    The tool finds the missing files in a geo-replication slave volume.
    The tool crawls backend .glusterfs of the brickpath, which is passed
    as a parameter and stats each entry on slave volume mount to check
    the presence of file. The mount used is aux-gfid-mount, hence no path
    conversion is required and is fast. The tool needs to be run on every
    node in cluster for each brickpath of geo-rep master volume to find
    missing files on slave volume. The tool is generic enough and can be
    used in non geo-replication context as well.
    
    Most of the crawler code is leverged from Avati's xfind and is modified
    to crawl only .glusterfs (https://github.com/avati/xsync)
    
    Thanks Aravinda for scripts to convert gfid to path.
    
    Change-Id: I84deaaaf638f7c571ff1319b67a3440fe27da810
    BUG: 1187140
    Signed-off-by: Aravinda VK <avishwan>
    Signed-off-by: Kotresh HR <khiremat>
    Reviewed-on: http://review.gluster.org/9503
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 14 Niels de Vos 2015-05-14 17:26:25 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 15 Niels de Vos 2015-05-14 17:28:20 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 16 Niels de Vos 2015-05-14 17:35:15 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user