Bug 1374153 - [RFE] History Crawl performance improvement
Summary: [RFE] History Crawl performance improvement
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: geo-replication
Version: 3.9
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Aravinda VK
QA Contact:
URL:
Whiteboard:
Depends On: 1364420 1503170
Blocks: 1364421 1365119
TreeView+ depends on / blocked
 
Reported: 2016-09-08 06:12 UTC by Aravinda VK
Modified: 2017-10-17 13:25 UTC (History)
1 user (show)

Fixed In Version: glusterfs-3.9.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1364420
Environment:
Last Closed: 2016-12-06 05:59:18 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Aravinda VK 2016-09-08 06:12:34 UTC
+++ This bug was initially created as a clone of Bug #1364420 +++

Description of problem:
If History changelogs backlog to be processed is more then Geo-rep takes lot of time to reach to current state because of rsync retries.

Issue and possible solution is discussed in upstream
http://www.gluster.org/pipermail/gluster-devel/2016-August/050372.html

--- Additional comment from Aravinda VK on 2016-08-08 09:00:13 EDT ---



--- Additional comment from Vijay Bellur on 2016-08-08 09:01:47 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#1) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-10 10:10:05 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#2) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-11 05:03:28 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#3) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-11 05:54:40 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#4) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-18 10:56:49 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#5) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-20 09:53:37 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#6) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-20 12:05:27 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#7) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-24 01:48:06 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#8) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-24 12:55:24 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#9) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-25 08:06:27 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#10) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-26 02:06:54 EDT ---

REVIEW: http://review.gluster.org/15110 (geo-rep: Post process Data and Meta Changelogs) posted (#11) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-26 13:46:03 EDT ---

COMMIT: http://review.gluster.org/15110 committed in master by Aravinda VK (avishwan@redhat.com) 
------
commit 6c283f107b646405936520e2549510115bf2ef64
Author: Aravinda VK <avishwan@redhat.com>
Date:   Mon Aug 8 17:02:37 2016 +0530

    geo-rep: Post process Data and Meta Changelogs
    
    With this patch, Data and Meta GFIDs are post processed. If Changelog has
    UNLINK entry then remove from Data and Meta GFIDs list(If stat on GFID is
    ENOENT in Master).
    
    While processing Changelogs,
    
    - Collect all the data and meta operations in a temporary database
    - Delete all Data and Meta GFIDs which are already unlinked as per Changelogs
      (unlink only if stat on GFID is ENOENT)
    - Process all Entry operations as usual
    - Process data and meta operations in batch(Fetch from Db in batch)
    - Data sync is again batched based on number of changelogs(Default 1day
      changelogs). Once the sync is complete, Update last Changelog's time as last_synced
      time as usual.
    
    Additionally maintain entry_stime on Brick root, ignore Entry ops if changelog
    suffix time is less than entry_stime. If data stime is more than entry_stime,
    this can happen only when passive worker updates stime by itself by getting
    mount point stime. Use entry_stime = data_stime in this case.
    
    New configurations:
    
    max-rsync-retries - Default Value is 10
    max-data-changelogs-in-batch - Max number of changelogs to be considered in a
    batch for syncing. Default value is 5760(4 changelogs per min * 60 min *
    24 hours)
    max-history-changelogs-in-batch - Max number of history changelogs to be
    processed at once. Default value 86400(4 changelogs per min * 60 min * 24
    hours * 15 days)
    
    BUG: 1364420
    Change-Id: I7b665895bf4806035c2a8573d361257cbadbea17
    Signed-off-by: Aravinda VK <avishwan@redhat.com>
    Reviewed-on: http://review.gluster.org/15110
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Kotresh HR <khiremat@redhat.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>

--- Additional comment from Worker Ant on 2016-08-31 06:12:11 EDT ---

REVIEW: http://review.gluster.org/15371 (geo-rep: Fix History post process) posted (#1) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Aravinda VK on 2016-08-31 07:41:50 EDT ---

Performance not gaining much as expected. Moving back to post to send fix to remove db changes.

--- Additional comment from Worker Ant on 2016-09-01 02:04:52 EDT ---

REVIEW: http://review.gluster.org/15371 (geo-rep: Fix History post process) posted (#2) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Worker Ant on 2016-09-05 01:54:20 EDT ---

REVIEW: http://review.gluster.org/15371 (geo-rep: Fix History post process) posted (#3) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Worker Ant on 2016-09-08 02:11:22 EDT ---

COMMIT: http://review.gluster.org/15371 committed in master by Aravinda VK (avishwan@redhat.com) 
------
commit 5de500cd0116796ff797099c60d33258bd48ce3c
Author: Aravinda VK <avishwan@redhat.com>
Date:   Wed Aug 31 11:53:06 2016 +0530

    geo-rep: Fix History post process
    
    This patch removes changelogsdb part of post processing since
    not got much performance advantage as expected.
    
    Entry stime and other logging improvements retained.
    
    BUG: 1364420
    Change-Id: Ib99d23f09d96c14bc28225b47d9134260f5551bf
    Signed-off-by: Aravinda VK <avishwan@redhat.com>
    Reviewed-on: http://review.gluster.org/15371
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Kotresh HR <khiremat@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.org>

Comment 1 Worker Ant 2016-09-08 06:14:06 UTC
REVIEW: http://review.gluster.org/15424 (geo-rep: Fix History post process) posted (#1) for review on release-3.9 by Aravinda VK (avishwan@redhat.com)

Comment 2 Worker Ant 2016-09-08 16:14:11 UTC
COMMIT: http://review.gluster.org/15424 committed in release-3.9 by Aravinda VK (avishwan@redhat.com) 
------
commit 001b2080de3793d02d988825db97fd2d4cba4829
Author: Aravinda VK <avishwan@redhat.com>
Date:   Wed Aug 31 11:53:06 2016 +0530

    geo-rep: Fix History post process
    
    This patch removes changelogsdb part of post processing since
    not got much performance advantage as expected.
    
    Entry stime and other logging improvements retained.
    
    > Reviewed-on: http://review.gluster.org/15371
    > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    > Reviewed-by: Kotresh HR <khiremat@redhat.com>
    > Smoke: Gluster Build System <jenkins@build.gluster.org>
    
    BUG: 1374153
    Change-Id: Ib99d23f09d96c14bc28225b47d9134260f5551bf
    Signed-off-by: Aravinda VK <avishwan@redhat.com>
    
    (cherry picked from commit 5de500cd0116796ff797099c60d33258bd48ce3c)
    
    Change-Id: If4ac94de557dd79a1e7c43e01ea8a1d6566569f4
    Reviewed-on: http://review.gluster.org/15424
    Tested-by: Aravinda VK <avishwan@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Kotresh HR <khiremat@redhat.com>

Comment 3 Aravinda VK 2016-10-27 05:21:59 UTC
glusterfs-3.9.0rc2 is released[1] and packages are available for different distributions[2] to test.

[1] http://www.gluster.org/pipermail/maintainers/2016-October/001601.html
[2] http://www.gluster.org/pipermail/maintainers/2016-October/001605.html and http://www.gluster.org/pipermail/maintainers/2016-October/001606.html

Comment 4 Aravinda VK 2016-12-06 05:59:18 UTC
Gluster 3.9 GA is released http://blog.gluster.org/2016/11/announcing-gluster-3-9/


Note You need to log in before you can comment on or make changes to this bug.