Description of problem: I had regression run of nfs-ganesha on 3.4 BU2 . During run of large IO I observed the major performance drop in random. write. Performance drop by 82 % compare to 3.4 BU1. I had two round of run and, in both the run I have seen performance drop. Details of test is as below H/W and S/W info : - 6 Node cluster IO tool : - IOZONE Volume Type : - Replica3 Linux OS : - RHEL 7.6 Volume details : ============== [root@gqas001 ~]# gluster v info testvol Volume Name: testvol Type: Distributed-Replicate Volume ID: c4cc8a4e-ea78-42f6-a538-e0ade46775da Status: Started Snapshot Count: 0 Number of Bricks: 24 x 3 = 72 Transport-type: tcp Bricks: Brick1: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick3: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick4: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick5: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick6: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick7: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick8: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick9: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick10: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick11: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick12: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick13: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick14: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick15: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick16: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick17: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick18: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick19: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick20: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick21: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick22: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick23: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick24: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick25: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick26: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick27: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick28: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick29: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick30: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick31: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick32: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick33: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick34: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick35: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick36: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick37: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick38: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick39: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick40: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick41: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick42: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick43: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick44: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick45: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick46: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick47: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick48: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick49: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick50: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick51: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick52: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick53: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick54: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick55: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick56: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick57: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick58: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick59: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick60: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick61: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick62: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick63: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick64: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick65: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick66: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick67: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick68: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick69: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick70: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick71: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick72: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Options Reconfigured: ganesha.enable: on server.event-threads: 4 client.event-threads: 4 cluster.lookup-optimize: on network.inode-lru-limit: 50000 performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.stat-prefetch: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on transport.address-family: inet nfs.disable: on performance.client-io-threads: off nfs-ganesha: enable cluster.enable-shared-storage: enable [root@gqas001 ~]# =============== Gluster details : =============== [root@gqas001 ~]# rpm -qa | grep gluster glusterfs-api-3.12.2-28.el7rhgs.x86_64 glusterfs-ganesha-3.12.2-28.el7rhgs.x86_64 tmp-rhs-tests-beaker-rhs-gluster-qe-libs-dev-bturner-3.0-0.noarch glusterfs-libs-3.12.2-28.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-28.el7rhgs.x86_64 glusterfs-debuginfo-3.12.2-18.el7.x86_64 glusterfs-3.12.2-28.el7rhgs.x86_64 glusterfs-cli-3.12.2-28.el7rhgs.x86_64 glusterfs-rdma-3.12.2-28.el7rhgs.x86_64 glusterfs-client-xlators-3.12.2-28.el7rhgs.x86_64 glusterfs-server-3.12.2-28.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-10.el7rhgs.x86_64 python2-gluster-3.12.2-28.el7rhgs.x86_64 samba-vfs-glusterfs-4.8.5-101.el7rhgs.x86_64 glusterfs-fuse-3.12.2-28.el7rhgs.x86_64 [root@gqas001 ~]# ================ SOS report and ganesha logs are under , - qe.eng.blr.redhat.com:/var/www/html/sosreports/bug.smali Please let me know if you need more information on this bug.