Bug 1642849

Summary: [Perf] Performance Drop observed in NFS-GANESHA "Delete-Rename" with FIPs enable
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sachin P Mali <smali>
Component: nfs-ganeshaAssignee: Xavi Hernandez <jahernan>
Status: CLOSED DEFERRED QA Contact: Rahul Hinduja <rhinduja>
Severity: low Docs Contact:
Priority: low    
Version: rhgs-3.4CC: amukherj, arjsharm, atumball, dang, ffilz, grajoria, jthottan, mbenjamin, pasik, rhinduja, rhs-bugs, sanandpa, sankarshan, skoduri, storage-qa-internal
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-16 04:22:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1651050    
Bug Blocks:    

Description Sachin P Mali 2018-10-25 08:28:04 UTC
Description of problem:

I ran the small-file performance test on NFS-Ganesha v4.0  mounted volume. After test, I observed performance drop in Delete-Rename FOPs by  41%. Test server has FIPs enable and 3.4 BU1 installed.
Please let me know if need any more information on it.
Gluster version : 
###############
[root@gqas001 ~]# rpm -qa | grep gluster
glusterfs-fuse-3.12.2-23.el7rhgs.x86_64
glusterfs-rdma-3.12.2-23.el7rhgs.x86_64
glusterfs-libs-3.12.2-23.el7rhgs.x86_64
glusterfs-3.12.2-23.el7rhgs.x86_64
glusterfs-cli-3.12.2-23.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-23.el7rhgs.x86_64
tmp-rhs-tests-beaker-rhs-gluster-qe-libs-dev-bturner-3.0-0.noarch
glusterfs-client-xlators-3.12.2-23.el7rhgs.x86_64
glusterfs-debuginfo-3.12.2-18.el7.x86_64
glusterfs-server-3.12.2-23.el7rhgs.x86_64
python2-gluster-3.12.2-23.el7rhgs.x86_64
glusterfs-api-3.12.2-23.el7rhgs.x86_64
[root@gqas001 ~]# 
###############
Options :
###############
performance.cache-samba-metadata: on
server.event-threads: 4
client.event-threads: 4
cluster.lookup-optimize: on
network.inode-lru-limit: 90000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
transport.address-family: inet
nfs.disable: off
performance.client-io-threads: off
################
Volume Type : Replica-3
Cluster : 6 servers and 6 clients

FIPs: Enable : 

################
[root@gqas001 ~]# cat /boot/grub2/grub.cfg | grep -i fip
	linux16 /vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/rhel_gqas001-root ro crashkernel=auto rd.lvm.lv=rhel_gqas001/root rd.lvm.lv=rhel_gqas001/swap selinux=0 console=ttyS1,115200 LANG=en_US.UTF-8 fips=1 boot=UUID=03794366-b295-4b3a-91c4-02c9553bab56
[root@gqas001 ~]# 
################
RHEL version : RHEL 7.6

Note :
   - I also have seed performance drop in FUSE, Bug 1641908 has filed for it.

Details of performance number are in below link,
https://docs.google.com/spreadsheets/d/1B2obaj3w85O2xtzqc_anyJoDy9Nnejg3jvBl3mWivMc/edit#gid=0