Bug 1713890

Summary: [Perf] Rename regressing by 10% in replica3 volume over fuse
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sweta Anandpara <sanandpa>
Component: coreAssignee: Kotresh HR <khiremat>
Status: CLOSED ERRATA QA Contact: Vivek Das <vdas>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.5CC: amukherj, jahernan, khiremat, lbailey, nbalacha, nchilaka, pkarampu, rgowdapp, rhs-bugs, sheggodu, storage-qa-internal, vdas
Target Milestone: ---Keywords: Performance, Regression
Target Release: RHGS 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0-10 Doc Type: Bug Fix
Doc Text:
The ctime feature is now disabled by default because enabling both ctime and gfid2path by default slowed rename operations by approximately ten percent.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-30 12:21:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1696809, 1727287    

Description Sweta Anandpara 2019-05-25 12:38:13 UTC 
Description of problem:
========================
Perf tests were run on a 6node cluster, having 12 bricks each, hosting a 24*3 volume 'testvol'. The volume was mounted over fuse, and I/O was run from 6 clients on 5000 files of size 64Kb each, via 8 threads. 
All the readings noted for data and metadata OPs were compared with the last GAed build RHGS 3.4 Batch Update 4. While all of them are in the acceptable discrepancy limit, rename OP is showing a drop of 10%. 

It is expected for some deviation to be present, and hence a number is arrived after taking an average of multiple runs. 


Version-Release number of selected component (if applicable):
============================================================
glusterfs-6.0-3


How reproducible:
=================
Always


Steps to Reproduce:
===================
1. Create a 24*3 volume on a 6node cluster having the build 
2. Run small file script from client, over fuse
3. Compare the files/sec reading with the baseline of RHGS 3.4 Batch Update 4

Actual results:
===============
Rename shows a drop of 10%

Expected results:
===================
No drop >5% should be seen in any of the data/metadata OPs

Additional info:
=================

Please find the comparison below for all the OPs over multiple builds. The same and more readings can be viewed @https://docs.google.com/spreadsheets/d/1r-peEBfHjds27ISwuikWgPJ4djAGQqqZu397wGrWtE4/edit?usp=sharing

Sosreports, smallfile script used, perf run logs, volume profile info can be found at: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/swetas/<bugnumber>


Release 	RHGS 3.4 BU4	       RHGS 3.5 - Cluster1		
-----------------------------------------------------------	

Build         "glusterfs-3.12.2-47"	glusterfs-6.0-1	         glusterfs-6.0-2	       glusterfs-6.0-3	          3.5  v/s  3.4.4
------------------------------------------------------------------------------------------------------------------------------------------
Creates	          2535	                2645.25	                 2623.25	               2619.75	                  3.34%
ls-l	          7548	                8822.5	                 8635	                       8933.13	                  18.35%
chmod	          1759	                1765.5	                 1727.25	               1766.75	                  0.44%
stat	          3163.5	        3108.75	                 3079.25	               3113.63	                 -1.58%
Reads	          3391.2	        3366.4	                 3324.6	                       3381.2	                 -0.29%
Appends	          1382.25	        1366.75	                 1344.25	               1362.13	                 -1.46%
Rename	          256.75	        227.75	                 239.25	                       230	                 -10.42%
Delete-Renamed	  2541.2	        2601.6	                 2569.6	                       2639.2	                  3.86%
Mkdir	          743.4	                748.2	                 747.6	                       762.1	                  2.52%
Rmdir	          474.8	                464.8	                 469.4	                       471.2	                  -0.76%


[root@gqas010 ~]# gluster v info
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 70cf29b4-08ed-48cf-8ef2-35834f841be9
Status: Started
Snapshot Count: 0
Number of Bricks: 24 x 3 = 72
Transport-type: tcp
Bricks:
Brick1: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick3: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick4: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick5: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick6: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick7: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick8: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick9: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick10: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick11: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick12: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick13: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick14: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick15: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick16: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick17: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick18: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick19: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick20: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick21: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick22: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick23: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick24: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick25: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick26: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick27: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick28: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick29: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick30: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick31: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick32: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick33: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick34: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick35: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick36: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick37: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick38: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick39: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick40: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick41: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick42: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick43: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick44: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick45: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick46: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick47: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick48: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick49: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick50: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick51: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick52: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick53: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick54: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick55: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick56: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick57: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick58: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick59: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick60: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick61: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick62: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick63: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick64: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick65: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick66: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick67: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick68: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick69: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick70: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick71: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick72: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Options Reconfigured:
server.event-threads: 4
client.event-threads: 4
cluster.lookup-optimize: on
network.inode-lru-limit: 200000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
transport.address-family: inet
nfs.disable: off
performance.client-io-threads: off
[root@gqas010 ~]#
Comment 15 Vivek Das 2019-10-23 11:49:04 UTC 
Marking this bug as Verified based on Comment 14
Comment 17 errata-xmlrpc 2019-10-30 12:21:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249