Bug 1651040

Summary: Add support for multi-threaded fuse reader threads for better performance
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Krutika Dhananjay <kdhananj>
Component: fuseAssignee: Krutika Dhananjay <kdhananj>
Status: CLOSED ERRATA QA Contact: Upasana <ubansal>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.4CC: asriram, atumball, kdhananj, nchilaka, rgowdapp, rhs-bugs, sankarshan, sheggodu, srmukher, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.4.z Batch Update 3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-33 Doc Type: Enhancement
Doc Text:
Red Hat Gluster Storage introduces a feature, reader-thread-count, for multi-threaded FUSE reader threads, which imports parallel requests on a FUSE mount and handles them through multiple threads, proffering better I/O performance.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-04 07:41:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Krutika Dhananjay 2018-11-19 02:29:52 UTC
Description of problem:

When Manoj Pillai, Raghavendra G and I were working on enhancing gluster read/write performance on NVMe backend, at one point it was observed that the fuse reader thread hit ~97% utilization even with client-io-threads enabled. At this point, single-threaded fuse reader became the bottleneck. The only option left was to scale the number of fuse threads. With this, we saw 8K iops improvement with 4 reader threads. Refer to https://goo.gl/AubdwP and/or https://goo.gl/6VbqRA for more information on actual tests performed and the results.

Clone of https://github.com/gluster/glusterfs/issues/412

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 12 Nag Pavan Chilakam 2019-01-02 09:13:59 UTC
sorry, should have asked the questions at one go, sorry for the inconvinience.
Below are some more questions


1) is there any other options that needs to be set, I believe we need to change io-threads value too see the optimal value. Any insights here will be really helpful
2) ideally this the fuse reader threads must be handled internally as a QOS,based on the load, instead of leaving it to the end user to set the value, as the end user may not be educated on the purpose and also can't keep remounting again.  Is there a bug for this(could be an RFE)?

Comment 13 Krutika Dhananjay 2019-01-03 05:45:23 UTC
(In reply to nchilaka from comment #12)
> sorry, should have asked the questions at one go, sorry for the
> inconvinience.
> Below are some more questions
> 
> 
> 1) is there any other options that needs to be set, I believe we need to
> change io-threads value too see the optimal value. Any insights here will be
> really helpful

Wasn't necessary in our experience at least. You can have the client-io-threads disabled since parallel requests can now be handled by multiple fuse threads. No change in the brick stack.


> 2) ideally this the fuse reader threads must be handled internally as a
> QOS,based on the load, instead of leaving it to the end user to set the
> value, as the end user may not be educated on the purpose and also can't
> keep remounting again.  Is there a bug for this(could be an RFE)?

Yes, there is - https://github.com/gluster/glusterfs/issues/406

-Krutika

Comment 23 Srijita Mukherjee 2019-01-20 19:19:20 UTC
The doc text has been updated. Kindly review the technical accuracy.

Comment 24 Krutika Dhananjay 2019-01-21 05:38:52 UTC
Copy-pasting the provided doc text here:

"Red Hat Gluster Storage introduces a feature for multi-threaded FUSE reader threads, which imports parallel requests on a FUSE mount and handled them by multiple threads, proffering better I/O performance."

Two things:
1. The verb tense used in "which imports parallel requests on a FUSE mount and ***handled*** them by multiple threads" doesn't sound right.
2. Are we not going to mention the name of the option and how to configure it as well, as part of the doc text?

-Krutika

Comment 25 Srijita Mukherjee 2019-01-21 06:37:41 UTC
(In reply to Krutika Dhananjay from comment #24)
> Copy-pasting the provided doc text here:
> 
> "Red Hat Gluster Storage introduces a feature for multi-threaded FUSE reader
> threads, which imports parallel requests on a FUSE mount and handled them by
> multiple threads, proffering better I/O performance."
> 
> Two things:
> 1. The verb tense used in "which imports parallel requests on a FUSE mount
> and ***handled*** them by multiple threads" doesn't sound right.
>> Have made the necessary changes. 
> 2. Are we not going to mention the name of the option and how to configure
> it as well, as part of the doc text?
>> The doc bug will take care of the configuration part. I have added the name of the option in the doc text.    
> -Krutika

>> Let me know if anything else needs to be added here.

Comment 26 Krutika Dhananjay 2019-01-21 06:50:02 UTC
(In reply to Srijita Mukherjee from comment #25)
> (In reply to Krutika Dhananjay from comment #24)
> > Copy-pasting the provided doc text here:
> > 
> > "Red Hat Gluster Storage introduces a feature for multi-threaded FUSE reader
> > threads, which imports parallel requests on a FUSE mount and handled them by
> > multiple threads, proffering better I/O performance."
> > 
> > Two things:
> > 1. The verb tense used in "which imports parallel requests on a FUSE mount
> > and ***handled*** them by multiple threads" doesn't sound right.
> >> Have made the necessary changes. 
> > 2. Are we not going to mention the name of the option and how to configure
> > it as well, as part of the doc text?
> >> The doc bug will take care of the configuration part. I have added the name of the option in the doc text.    
> > -Krutika
> 
> >> Let me know if anything else needs to be added here.

Looks good to me.

-Krutika

Comment 28 errata-xmlrpc 2019-02-04 07:41:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0263