Bug 1467094

Summary: [Perf] : mkdirs and rmdirs have slowed down by ~48%
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Ambarish <asoman>
Component: nfs-ganeshaAssignee: Kaleb KEITHLEY <kkeithle>
Status: CLOSED NOTABUG QA Contact: Ambarish <asoman>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.3CC: bturner, dang, ffilz, jthottan, kkeithle, mbenjamin, rhinduja, rhs-bugs, skoduri, storage-qa-internal
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-05 03:00:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ambarish 2017-07-02 17:36:17 UTC 
Description of problem:
-----------------------


Baseline : RHGS 3.2 + NFS Ganesha 2.4.1-9

Currently testing Ganesha 2.4.4-10 and Gluster 3.8.4-32 build.

*mkdir* :

3.2 : 867 files/sec

Latest 3.3 bits : 490 files/sec

Regression : ~43%

*rmdir* :

3.2 : 638 files/sec

Latest 3.3 bits : 314 files/sec

Regression : ~50%

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

nfs-ganesha-2.4.4-10.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-32.el7rhgs.x86_64

How reproducible:
-----------------

Always



Additional info:
----------------

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: a50c906d-1c2f-4942-b31b-4ad2a2223b3d
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas001.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas010.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
client.event-threads: 4
server.event-threads: 4
cluster.lookup-optimize: on
ganesha.enable: on
features.cache-invalidation: on
server.allow-insecure: on
performance.stat-prefetch: off
transport.address-family: inet
nfs.disable: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable
Comment 5 Ambarish 2017-07-05 03:00:15 UTC 
False Alarm!

This seems to occur since the volume got mounted by v4.1(https://bugzilla.redhat.com/show_bug.cgi?id=1375259) and I missed that while opening this BZ.

Mounting via v4.0,the perf regression for this test is within limits that my tests are allowed to have between releases/dev builds.

Closing as Nab,and appologies for the unnecesssary noise.
Comment 6 Frank Filz 2017-07-05 14:01:10 UTC 
(In reply to Ambarish from comment #5)
> False Alarm!
> 
> This seems to occur since the volume got mounted by
> v4.1(https://bugzilla.redhat.com/show_bug.cgi?id=1375259) and I missed that
> while opening this BZ.
> 
> Mounting via v4.0,the perf regression for this test is within limits that my
> tests are allowed to have between releases/dev builds.
> 
> Closing as Nab,and appologies for the unnecesssary noise.

Yet still interesting. If v4.1 causes this much of a performance degradation, we will definitely need to look into it.

Curious what the perf difference is with kernel nfsd.
Comment 7 Ambarish 2017-08-10 07:19:15 UTC 
Upstream BZ to track this : https://bugzilla.redhat.com/show_bug.cgi?id=1480098