Bug 1652720 - [Perf] Performance Drop by 82% observed in NFS-GANESHA random. write in large file
Summary: [Perf] Performance Drop by 82% observed in NFS-GANESHA random. write in large...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: nfs-ganesha
Version: rhgs-3.4
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: ---
Assignee: Kaleb KEITHLEY
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-22 17:28 UTC by Sachin P Mali
Modified: 2018-11-30 17:56 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-30 17:56:25 UTC
Embargoed:


Attachments (Terms of Use)

Description Sachin P Mali 2018-11-22 17:28:07 UTC
Description of problem:
 I had  regression run of  nfs-ganesha on 3.4 BU2 . During run of large IO I observed the major performance drop in random. write. Performance drop by 82 % compare to 3.4 BU1. I had two round of run and, in both the run I have seen performance drop. Details of test is as below

H/W and S/W info : 
- 6 Node cluster 
IO tool : 
- IOZONE
Volume Type : 
- Replica3
Linux OS : 
- RHEL 7.6

Volume details : 

==============
[root@gqas001 ~]# gluster v info testvol
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: c4cc8a4e-ea78-42f6-a538-e0ade46775da
Status: Started
Snapshot Count: 0
Number of Bricks: 24 x 3 = 72
Transport-type: tcp
Bricks:
Brick1: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick3: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick4: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick5: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick6: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol
Brick7: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick8: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick9: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick10: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick11: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick12: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol
Brick13: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick14: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick15: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick16: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick17: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick18: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol
Brick19: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick20: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick21: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick22: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick23: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick24: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol
Brick25: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick26: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick27: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick28: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick29: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick30: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol
Brick31: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick32: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick33: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick34: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick35: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick36: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol
Brick37: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick38: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick39: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick40: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick41: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick42: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol
Brick43: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick44: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick45: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick46: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick47: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick48: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol
Brick49: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick50: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick51: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick52: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick53: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick54: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol
Brick55: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick56: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick57: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick58: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick59: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick60: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol
Brick61: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick62: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick63: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick64: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick65: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick66: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol
Brick67: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick68: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick69: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick70: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick71: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Brick72: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol
Options Reconfigured:
ganesha.enable: on
server.event-threads: 4
client.event-threads: 4
cluster.lookup-optimize: on
network.inode-lru-limit: 50000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
nfs-ganesha: enable
cluster.enable-shared-storage: enable
[root@gqas001 ~]#
===============

Gluster details : 
===============
[root@gqas001 ~]# rpm -qa | grep gluster
glusterfs-api-3.12.2-28.el7rhgs.x86_64
glusterfs-ganesha-3.12.2-28.el7rhgs.x86_64
tmp-rhs-tests-beaker-rhs-gluster-qe-libs-dev-bturner-3.0-0.noarch
glusterfs-libs-3.12.2-28.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-28.el7rhgs.x86_64
glusterfs-debuginfo-3.12.2-18.el7.x86_64
glusterfs-3.12.2-28.el7rhgs.x86_64
glusterfs-cli-3.12.2-28.el7rhgs.x86_64
glusterfs-rdma-3.12.2-28.el7rhgs.x86_64
glusterfs-client-xlators-3.12.2-28.el7rhgs.x86_64
glusterfs-server-3.12.2-28.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-10.el7rhgs.x86_64
python2-gluster-3.12.2-28.el7rhgs.x86_64
samba-vfs-glusterfs-4.8.5-101.el7rhgs.x86_64
glusterfs-fuse-3.12.2-28.el7rhgs.x86_64
[root@gqas001 ~]# 
================
SOS  report and ganesha logs are under ,
- qe.eng.blr.redhat.com:/var/www/html/sosreports/bug.smali

Please let me know if you need more information on this bug.


Note You need to log in before you can comment on or make changes to this bug.