Bug 1449638

Summary: Poor write speed performance of fio test on distributed-disperse volume
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Prashant Dhange <pdhange>
Component: disperseAssignee: Ashish Pandey <aspandey>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.2CC: amukherj, aspandey, pdhange, pkarampu, pmulay, psony, rhinduja, rhs-bugs, sheggodu, storage-qa-internal, ubansal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: rebase
Fixed In Version: glusterfs-3.12.2-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 06:32:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1417153, 1472361, 1503135    

Comment 41 Ashish Pandey 2017-07-26 10:14:33 UTC
Hi,

The fio issue has been fixed in https://github.com/axboe/fio/issues/376
I still need to verify it at my end.

Atin,
I think the content of the doc text will be different now.

Comment 54 Nag Pavan Chilakam 2018-05-29 16:13:41 UTC
on-qa validation:
based on  all above comments this is a problem with fio writing in 1Byte chunks
As this should be fixed in latest FIO, I am comparing the writes b/w fio.2.2.8(where the issue was seen) as with latest fio
also tested with 10MB file
below is the result

[root@dhcp35-72 dispersevol1]# rpm -qa|grep fio
fio-2.2.8-2.el7.x86_64
[root@dhcp35-72 dispersevol1]# time fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=test --filename=fiotest.fio --iodepth=16 --size=10M --readwrite=write
test: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
fio-2.2.8
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 10MB)
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/1144KB/0KB /s] [0/286/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=11112: Tue May 29 21:35:52 2018
  write: io=10240KB, bw=1160.4KB/s, iops=290, runt=  8825msec
  cpu          : usr=0.27%, sys=1.10%, ctx=2566, majf=0, minf=23
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2560/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: io=10240KB, aggrb=1160KB/s, minb=1160KB/s, maxb=1160KB/s, mint=8825msec, maxt=8825msec

real	0m19.064s
user	0m0.136s
sys	0m0.718s



########## with latest fio
[root@dhcp35-72 fio]# time fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=test --filename=fiotest.fio --iodepth=16 --size=10M --readwrite=write
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.1
Starting 1 process
test: Laying out IO file (1 file / 10MiB)
fio: native_fallocate call failed: Operation not supported
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=976KiB/s][r=0,w=244 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=11132: Tue May 29 21:41:55 2018
  write: IOPS=270, BW=1080KiB/s (1106kB/s)(10.0MiB/9479msec)
   bw (  KiB/s): min=  888, max= 1208, per=99.39%, avg=1073.44, stdev=92.80, samples=18
   iops        : min=  222, max=  302, avg=268.33, stdev=23.19, samples=18
  cpu          : usr=0.19%, sys=1.52%, ctx=2566, majf=0, minf=20
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,2560,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1080KiB/s (1106kB/s), 1080KiB/s-1080KiB/s (1106kB/s-1106kB/s), io=10.0MiB (10.5MB), run=9479-9479msec

real	0m10.252s
user	0m0.118s
sys	0m0.665s
[root@dhcp35-72 fio]# date
Tue May 29 21:41:57 IST 2018
[root@dhcp35-72 fio]# rpm -qa|grep fio
fio-3.1-2.el7.x86_64
[root@dhcp35-72 fio]# du -sh fiotest.fio 
10M	fiotest.fio
[root@dhcp35-72 fio]# 


as the performance has improved a lot , moving to verified


test on gnfs as reported by customer

Comment 55 Atin Mukherjee 2018-09-03 14:13:50 UTC
Srijita brought this bug to my notice where this bug is actually marked as Verified

Comment 56 errata-xmlrpc 2018-09-04 06:32:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607