Description of problem: ======================= Over multiple runs done in NFSv4.0 protocol, a lot of inconsistency was noticed in results of varios file ops and metadata ops. An average was taken over 3 runs and 'append' op was found to regress by about 12%. Will continue subsequent runs to check if there is a positive trend noticed.. Will keep this bug updated with results. Version-Release number of selected component (if applicable): ============================================================ glusterfs-6.0-2 How reproducible: ================= Inconsistent results. Average results have been taken over 3 runs. Steps to Reproduce: =================== 1. On a 6 node cluster with 12 bricks each, create a 24*3 volume 'testvol' 2. Run small file tool for creates, read, append, mkdir, rmdir. These ops were chosen as they figured in border range of ~10% in the previous runs. 3. Compare the numbers with baseline of RHGS 3.4 Update 4 Actual results: =============== Drop of performance seen in append FOP to about 12% Expected results: ================== Regression (drop) if seen should not be >10% Additional info: ================ [root@gqas001 ~]# gluster volume info Volume Name: gluster_shared_storage Type: Replicate Volume ID: 64262ff1-eb76-4381-a05f-defa181618b3 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gqas004.sbu.lab.eng.bos.redhat.com:/var/lib/glusterd/ss_brick Brick2: gqas014.sbu.lab.eng.bos.redhat.com:/var/lib/glusterd/ss_brick Brick3: gqas001.sbu.lab.eng.bos.redhat.com:/var/lib/glusterd/ss_brick Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off nfs-ganesha: enable cluster.enable-shared-storage: enable Volume Name: testvol Type: Distributed-Replicate Volume ID: fc0a47b0-138b-4913-8685-fcb1cd305475 Status: Started Snapshot Count: 0 Number of Bricks: 24 x 3 = 72 Transport-type: tcp Bricks: Brick1: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick3: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick4: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick5: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick6: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick1/testvol Brick7: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick8: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick9: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick10: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick11: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick12: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick2/testvol Brick13: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick14: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick15: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick16: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick17: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick18: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick3/testvol Brick19: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick20: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick21: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick22: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick23: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick24: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick4/testvol Brick25: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick26: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick27: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick28: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick29: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick30: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick5/testvol Brick31: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick32: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick33: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick34: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick35: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick36: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick6/testvol Brick37: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick38: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick39: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick40: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick41: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick42: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick7/testvol Brick43: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick44: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick45: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick46: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick47: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick48: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick8/testvol Brick49: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick50: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick51: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick52: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick53: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick54: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick9/testvol Brick55: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick56: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick57: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick58: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick59: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick60: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick10/testvol Brick61: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick62: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick63: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick64: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick65: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick66: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick11/testvol Brick67: gqas001.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick68: gqas004.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick69: gqas009.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick70: gqas010.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick71: gqas012.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Brick72: gqas014.sbu.lab.eng.bos.redhat.com:/gluster/brick12/testvol Options Reconfigured: ganesha.enable: on server.event-threads: 4 client.event-threads: 4 cluster.lookup-optimize: on network.inode-lru-limit: 200000 performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.stat-prefetch: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on transport.address-family: inet nfs.disable: off performance.client-io-threads: off nfs-ganesha: enable cluster.enable-shared-storage: enable [root@gqas001 ~]# gluster pool list UUID Hostname State 83705b3e-b1c6-4d64-bee0-1e55425b263b gqas004.sbu.lab.eng.bos.redhat.com Connected b24bb0c4-3c60-42af-bff4-473b48b6b294 gqas014.sbu.lab.eng.bos.redhat.com Connected dab8813c-250b-4858-b57a-4bcb403a6ead gqas010.sbu.lab.eng.bos.redhat.com Connected 1a743b4d-ee15-4462-bdea-f1ea2746d4ab gqas012.sbu.lab.eng.bos.redhat.com Connected fbfde667-b292-4ede-a0ff-58841c1432a0 gqas009.sbu.lab.eng.bos.redhat.com Connected 1673bf7e-d3fd-440b-a844-8128f0deb500 localhost Connected [root@gqas001 ~]# [root@gqas001 ~]# gluster v get all all Option Value ------ ----- cluster.server-quorum-ratio 51 cluster.enable-shared-storage enable cluster.op-version 70000 cluster.max-op-version 70000 cluster.brick-multiplex disable cluster.max-bricks-per-process 250 glusterd.vol_count_per_thread 100 cluster.daemon-log-level INFO [root@gqas001 ~]# [root@gqas001 ~]# [root@gqas001 ~]# gluster v status Status of volume: gluster_shared_storage Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gqas004.sbu.lab.eng.bos.redhat.com:/v ar/lib/glusterd/ss_brick 49152 0 Y 23705 Brick gqas014.sbu.lab.eng.bos.redhat.com:/v ar/lib/glusterd/ss_brick 49152 0 Y 24030 Brick gqas001.sbu.lab.eng.bos.redhat.com:/v ar/lib/glusterd/ss_brick 49152 0 Y 26815 Self-heal Daemon on localhost N/A N/A Y 26836 Self-heal Daemon on gqas010.sbu.lab.eng.bos .redhat.com N/A N/A Y 23412 Self-heal Daemon on gqas004.sbu.lab.eng.bos .redhat.com N/A N/A Y 23726 Self-heal Daemon on gqas014.sbu.lab.eng.bos .redhat.com N/A N/A Y 24051 Self-heal Daemon on gqas012.sbu.lab.eng.bos .redhat.com N/A N/A Y 23727 Self-heal Daemon on gqas009.sbu.lab.eng.bos .redhat.com N/A N/A Y 23759 Task Status of Volume gluster_shared_storage ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: testvol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick1/testvol 49153 0 Y 2860 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick1/testvol 49153 0 Y 21914 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick1/testvol 49152 0 Y 21902 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick1/testvol 49152 0 Y 21546 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick1/testvol 49152 0 Y 21863 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick1/testvol 49153 0 Y 22262 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick2/testvol 49154 0 Y 2880 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick2/testvol 49154 0 Y 21934 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick2/testvol 49153 0 Y 21922 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick2/testvol 49153 0 Y 21566 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick2/testvol 49153 0 Y 21883 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick2/testvol 49154 0 Y 22282 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick3/testvol 49155 0 Y 2900 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick3/testvol 49155 0 Y 21954 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick3/testvol 49154 0 Y 21943 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick3/testvol 49154 0 Y 21587 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick3/testvol 49154 0 Y 21904 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick3/testvol 49155 0 Y 22303 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick4/testvol 49156 0 Y 2920 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick4/testvol 49156 0 Y 21974 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick4/testvol 49155 0 Y 21963 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick4/testvol 49155 0 Y 21607 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick4/testvol 49155 0 Y 21924 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick4/testvol 49156 0 Y 22323 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick5/testvol 49157 0 Y 2940 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick5/testvol 49157 0 Y 21994 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick5/testvol 49156 0 Y 21983 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick5/testvol 49156 0 Y 21627 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick5/testvol 49156 0 Y 21944 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick5/testvol 49157 0 Y 22343 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick6/testvol 49158 0 Y 2960 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick6/testvol 49158 0 Y 22014 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick6/testvol 49157 0 Y 22003 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick6/testvol 49157 0 Y 21647 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick6/testvol 49157 0 Y 21964 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick6/testvol 49158 0 Y 22363 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick7/testvol 49159 0 Y 2980 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick7/testvol 49159 0 Y 22034 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick7/testvol 49158 0 Y 22023 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick7/testvol 49158 0 Y 21667 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick7/testvol 49158 0 Y 21984 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick7/testvol 49159 0 Y 22383 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick8/testvol 49160 0 Y 3000 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick8/testvol 49160 0 Y 22054 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick8/testvol 49159 0 Y 22043 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick8/testvol 49159 0 Y 21687 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick8/testvol 49159 0 Y 22004 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick8/testvol 49160 0 Y 22403 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick9/testvol 49161 0 Y 3020 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick9/testvol 49161 0 Y 22074 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick9/testvol 49160 0 Y 22063 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick9/testvol 49160 0 Y 21707 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick9/testvol 49160 0 Y 22024 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick9/testvol 49161 0 Y 22423 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick10/testvol 49162 0 Y 3040 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick10/testvol 49162 0 Y 22094 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick10/testvol 49161 0 Y 22083 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick10/testvol 49161 0 Y 21727 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick10/testvol 49161 0 Y 22044 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick10/testvol 49162 0 Y 22443 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick11/testvol 49163 0 Y 3060 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick11/testvol 49163 0 Y 22114 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick11/testvol 49162 0 Y 22103 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick11/testvol 49162 0 Y 21747 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick11/testvol 49162 0 Y 22064 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick11/testvol 49163 0 Y 22463 Brick gqas001.sbu.lab.eng.bos.redhat.com:/g luster/brick12/testvol 49164 0 Y 3080 Brick gqas004.sbu.lab.eng.bos.redhat.com:/g luster/brick12/testvol 49164 0 Y 22134 Brick gqas009.sbu.lab.eng.bos.redhat.com:/g luster/brick12/testvol 49163 0 Y 22123 Brick gqas010.sbu.lab.eng.bos.redhat.com:/g luster/brick12/testvol 49163 0 Y 21767 Brick gqas012.sbu.lab.eng.bos.redhat.com:/g luster/brick12/testvol 49163 0 Y 22084 Brick gqas014.sbu.lab.eng.bos.redhat.com:/g luster/brick12/testvol 49164 0 Y 22483 Self-heal Daemon on localhost N/A N/A Y 26836 NFS Server on localhost N/A N/A N N/A Self-heal Daemon on gqas014.sbu.lab.eng.bos .redhat.com N/A N/A Y 24051 NFS Server on gqas014.sbu.lab.eng.bos.redha t.com N/A N/A N N/A Self-heal Daemon on gqas004.sbu.lab.eng.bos .redhat.com N/A N/A Y 23726 NFS Server on gqas004.sbu.lab.eng.bos.redha t.com N/A N/A N N/A Self-heal Daemon on gqas010.sbu.lab.eng.bos .redhat.com N/A N/A Y 23412 NFS Server on gqas010.sbu.lab.eng.bos.redha t.com N/A N/A N N/A Self-heal Daemon on gqas012.sbu.lab.eng.bos .redhat.com N/A N/A Y 23727 NFS Server on gqas012.sbu.lab.eng.bos.redha t.com N/A N/A N N/A Self-heal Daemon on gqas009.sbu.lab.eng.bos .redhat.com N/A N/A Y 23759 NFS Server on gqas009.sbu.lab.eng.bos.redha t.com N/A N/A N N/A Task Status of Volume testvol ------------------------------------------------------------------------------ There are no active volume tasks [root@gqas001 ~]# [root@gqas001 ~]# rpm -qa | grep gluster glusterfs-client-xlators-6.0-2.el7rhgs.x86_64 tmp-rhs-tests-beaker-rhs-gluster-qe-libs-dev-bturner-3.0-0.noarch python2-gluster-6.0-2.el7rhgs.x86_64 glusterfs-libs-6.0-2.el7rhgs.x86_64 glusterfs-api-6.0-2.el7rhgs.x86_64 glusterfs-geo-replication-6.0-2.el7rhgs.x86_64 glusterfs-6.0-2.el7rhgs.x86_64 glusterfs-cli-6.0-2.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-12.el7rhgs.x86_64 glusterfs-server-6.0-2.el7rhgs.x86_64 glusterfs-ganesha-6.0-2.el7rhgs.x86_64 glusterfs-fuse-6.0-2.el7rhgs.x86_64 [root@gqas001 ~]#