Bug 1383648

Summary: [Perf] : Inconsistent writes over Ganesha .
Product: Red Hat Gluster Storage Reporter: Ambarish <asoman>
Component: nfs-ganeshaAssignee: Girjesh Rajoria <grajoria>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Sachin P Mali <smali>
Severity: high Docs Contact:
Priority: medium    
Version: rhgs-3.2CC: ffilz, jthottan, kkeithle, pasik, rhinduja, rhs-bugs, skoduri, smali, storage-qa-internal
Target Milestone: ---Keywords: Performance, ZStream
Target Release: ---Flags: grajoria: needinfo? (smali)
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-29 13:30:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Ambarish 2016-10-11 11:10:52 UTC 
Description of problem:
-----------------------

Sequential write over Ganesha v4 mounts seems inconsistent.This is from 7 back to back runs :

Test 1 : 555309 KB/sec
Test 2 : 418156 kB/sec
Test 3 : 523303 kB/sec
Test 4 : 529145 kB/sec
Test 5 : 470132 kB/sec
Test 6 : 525562 kB/sec
Test 7 : 643508 kB/sec

Seq Reads were OK.

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

nfs-ganesha-2.4.0-2.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-2.el7rhgs.x86_64


How reproducible:
------------------

Always. 


Steps to Reproduce:
-------------------

Run iozone sequential writes a couple of times

Actual results:
---------------

Inconsistency in writes on v4 mounts.

Expected results:
-----------------

Consistent performance.


Additional info:
----------------

* Vol Type  : 2*2,Dist Rep

* Client and Server OS : RHEL 7

* Vol config : 

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: b93b99bd-d1d2-4236-98bc-08311f94e7dc
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas013.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas005.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas006.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas011.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
ganesha.enable: on
features.cache-invalidation: off
nfs.disable: on
performance.readdir-ahead: on
performance.stat-prefetch: off
server.allow-insecure: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable
[root@gqas013 tmp]#
Comment 2 Ambarish 2016-10-11 11:13:05 UTC 
This is the write throughout on v3 mounts , on back to back runs :

Test 1 -> 1175969.83 kB/sec
Test 2 -> 1221023.66 kB/sec
Test 3 -> 1117435.82 kB/sec
Test 4 -> 1226414.77 kB/sec
Comment 3 Ambarish 2016-10-11 11:14:44 UTC 
Created attachment 1209119 [details]
Server profiles for v4,with inconsistent writes

Attaching  server profiles for all the runs on v4.
Comment 5 Ambarish 2017-05-08 05:01:28 UTC 
This happens on v3/v4 both - especially random workloads.
Comment 7 Kaleb KEITHLEY 2017-08-23 12:43:32 UTC 
stretch goal for 3.4.0
Comment 11 Girjesh Rajoria 2018-11-20 10:32:21 UTC 
I'm not able to reproduce this issue on my setup. This seems to be fixed with https://bugzilla.redhat.com/show_bug.cgi?id=1577822 .
Comment 12 Girjesh Rajoria 2018-11-20 10:38:36 UTC 
Can you please revalidate it.