Bug 1753857

Summary: geo-rep: performance improvement while syncing heavy renames with existing destination
Product: [Community] GlusterFS Reporter: Sunny Kumar <sunkumar>
Component: geo-replicationAssignee: Sunny Kumar <sunkumar>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: mainlineCC: amukherj, avishwan, bugs, csaba, khiremat, kiyer, puebele, rhs-bugs, storage-qa-internal
Target Milestone: ---Keywords: Performance
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1726000
: 1755212 (view as bug list) Environment:
Last Closed: 2019-09-23 12:07:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1726000    
Bug Blocks: 1696809, 1755212    

Description Sunny Kumar 2019-09-20 05:16:54 UTC 
+++ This bug was initially created as a clone of Bug #1726000 +++

Description of problem:

The bz#1694820
addresses issue for RENAME issue when handling RENAME with existing destination. This fix requires some performance tuning considering this issue occurs in heavy rename workload.
 
Version-Release number of selected component (if applicable):
master

How reproducible:


Steps to Reproduce:

1. Run frequent RENAME on master mount and check for sync in slave.

Ex - while true; do uuid="`uuidgen`"; echo "some data" > "test$uuid"; mv "test$uuid" "test" -f; done 

Actual results:


Expected results:


Additional info:
Comment 1 Worker Ant 2019-09-20 05:30:34 UTC 
REVIEW: https://review.gluster.org/23459 (geo-rep: performance improvement while syncing renames with existing gfid) posted (#1) for review on master by Sunny Kumar
Comment 2 Worker Ant 2019-09-23 12:07:40 UTC 
REVIEW: https://review.gluster.org/23459 (geo-rep: performance improvement while syncing renames with existing gfid) merged (#2) on master by Kotresh HR
Comment 3 Sunny Kumar 2019-10-22 06:42:10 UTC 
*********************
Performance Stat:

For this bug we have performed a performance run with arbiter and replica volumes with root geo-rep sessions.

Setup details:
- Number of Master servers: 3
- Number of slave servers: 3
- Brick size: 50GB
- RAM: 8GB
- Number of clients used: 1 
- Type of volume used:
  1. arbiter 1x(2+1)
  2. replica 1x3
- Workload used: for i in `seq 1 1000000`;do uuid="`uuidgen`"; date > "test$uuid"; mv "test$uuid" "test" -f; done

The results are as follows:

1. With sync type as rsync:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Volume type             Configuration         number of creates and renames done             time taken without fix                     time taken with fix
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Arbiter                  1x(2+1)                           1000000                      14 hours, 22 minutes, 56 seconds              2 hours, 4 minutes, 6 seconds
Replicate                 1x3                              1000000                      20 hours, 13 minutes, 14 seconds              8 hours, 27 minutes, 2 seconds

2. With sync type as tarssh:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Volume type             Configuration         number of creates and renames done             time taken without fix                     time taken with fix
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Arbiter                  1x(2+1)                           1000000                      20 hours, 22 minutes, 56 seconds              8 hours, 34 minutes, 20 seconds
Replicate                 1x3                              1000000                      22 hours, 49 minutes, 29 seconds              10 hours, 7 minutes, 43 seconds