Bug 1359180 - Make client.io-threads enabled by default
Summary: Make client.io-threads enabled by default
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: RHGS 3.2.0
Assignee: Pranith Kumar K
QA Contact: Byreddy
URL:
Whiteboard:
Depends On: 1361678
Blocks: 1351503
TreeView+ depends on / blocked
 
Reported: 2016-07-22 12:42 UTC by Atin Mukherjee
Modified: 2018-02-14 19:24 UTC (History)
9 users (show)

Fixed In Version: glusterfs-3.8.4-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-23 05:39:51 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC
Red Hat Bugzilla 1349953 None None None Never

Internal Links: 1349953

Description Atin Mukherjee 2016-07-22 12:42:42 UTC
Description of problem:

Turning on client.io-threads improves performance for parallel I/O from single mount point on EC/dist-rep volumes. This request is to turn on the option by default.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Atin Mukherjee 2016-08-05 06:20:07 UTC
Upstream patch : http://review.gluster.org/#/c/15051/ is merged in mainline.

Comment 4 Ravishankar N 2016-09-16 04:06:53 UTC
Downstream patch: https://code.engineering.redhat.com/gerrit/#/c/84684/

Comment 5 Atin Mukherjee 2016-09-16 04:08:48 UTC
Downstream patch is merged now.

Comment 10 Byreddy 2016-09-30 10:25:01 UTC
Verified this using the build - glusterfs-3.8.4-2

i am seeing volume option performace.client-io-threads enabled by default.

The default value of performace.client-io-threads in 3.1.3 and 3.2 ( 3.8.4-2 ) is below.

In 3.1.3
========
]# gluster volume get Dis performance.client-io-threads
Option                                  Value                                   
------                                  -----                                   
performance.client-io-threads           off                                     
[root@ ~]# 


In 3.2 
======
~]# gluster volume get Replica performance.client-io-threads
Option                                  Value                                   
------                                  -----                                   
performance.client-io-threads           on                                      
[root@ ~]# 

Based on above details moving to verified state.

Comment 11 Manoj Pillai 2016-10-16 05:10:34 UTC
Quantifying the performance benefit of enabling client-io-threads on a 12x(4+2) volume (on 6 servers, with 12 drives per server in JBOD mode) with the following options set:

cluster.lookup-optimize: on
server.event-threads: 4
client.event-threads: 4

Test:
<body>
MY_MNT="/mnt/glustervol"

for ii in 1 4 8; do
    iozone -i 0 -w -+n -c -e -s 10g -r 64k -t ${ii}  -F ${MY_MNT}/f{1..8}.ioz
    rm -f ${MY_MNT}/f*.ioz
done
</body>

Result:

No. of threads | client-io-threads=off | client-io-threads=on
1              | 136125.80 kB/sec      | 140190.06 kB/sec
4              | 505460.07 kB/sec      | 495928.16 kB/sec
8              | 177709.80 kB/sec      | 785245.68 kB/sec

Comment 12 Manoj Pillai 2016-10-17 06:57:38 UTC
Ambarish, could you post a followup to comment #11 with results that you see on your setup for the same test? It will help with some of the discussions that are going on.

Note: these are single-client iozone tests, not distributed.

Comment 13 Ambarish 2016-10-17 07:30:34 UTC
Manoj,

While testing this on 3.1.3,I tested using single and multiple clients both.

If I test it in a distributed-multithreaded way,I do not see much of a boost.

On a single client,though it's a different story..I see almost 200% increase on seq writes (RHEL 7 servers,RHEL 6 clients,8 threads,1 client,64k rec size).

Are the results on a single client good enough to make this param "on" by default in production?

If yes,we might need to advertise/communicate the exact config/scenarios where we are seeing major perf boosts.

Comment 15 Manoj Pillai 2016-10-17 08:05:43 UTC
(In reply to Ambarish from comment #13)
> Manoj,
> 
> While testing this on 3.1.3,I tested using single and multiple clients both.
> 
> If I test it in a distributed-multithreaded way,I do not see much of a boost.
> 
> On a single client,though it's a different story..I see almost 200% increase
> on seq writes (RHEL 7 servers,RHEL 6 clients,8 threads,1 client,64k rec
> size).
> 
> Are the results on a single client good enough to make this param "on" by
> default in production?

Yes, I think so. These are all approximations of workloads customers might see in production.

> 
> If yes,we might need to advertise/communicate the exact config/scenarios
> where we are seeing major perf boosts.

The actual flow of requests for EC is somewhat complex from what I understand. Which thread(s) end up doing the heavy lifting aka encoding for writes depends on dynamic conditions -- sometimes it can be the fuse-reader thread (or client-io-threads, if enabled), sometimes it can be the client.event-threads. 

I think eventually we want to be able to give users a simple guideline: turn on client-io-threads if you're using EC. Or better still, make client-io-threads=on the default, as this bz does (but, from what I understand, we are going to reverse this for RHGS 3.2 because of bugs we have hit).

Comment 16 Pranith Kumar K 2016-10-17 08:28:35 UTC
(In reply to Manoj Pillai from comment #15)
> (In reply to Ambarish from comment #13)
> > Manoj,
> > 
> > While testing this on 3.1.3,I tested using single and multiple clients both.
> > 
> > If I test it in a distributed-multithreaded way,I do not see much of a boost.
> > 
> > On a single client,though it's a different story..I see almost 200% increase
> > on seq writes (RHEL 7 servers,RHEL 6 clients,8 threads,1 client,64k rec
> > size).
> > 
> > Are the results on a single client good enough to make this param "on" by
> > default in production?
> 
> Yes, I think so. These are all approximations of workloads customers might
> see in production.
> 
> > 
> > If yes,we might need to advertise/communicate the exact config/scenarios
> > where we are seeing major perf boosts.
> 
> The actual flow of requests for EC is somewhat complex from what I
> understand. Which thread(s) end up doing the heavy lifting aka encoding for
> writes depends on dynamic conditions -- sometimes it can be the fuse-reader
> thread (or client-io-threads, if enabled), sometimes it can be the
> client.event-threads. 
> 
> I think eventually we want to be able to give users a simple guideline: turn
> on client-io-threads if you're using EC. Or better still, make
> client-io-threads=on the default, as this bz does (but, from what I
> understand, we are going to reverse this for RHGS 3.2 because of bugs we
> have hit).

Nope, it is better to fix them :-)

Comment 17 Manoj Pillai 2016-10-17 18:19:42 UTC
(In reply to Ambarish from comment #13)
> Manoj,
> 
> While testing this on 3.1.3,I tested using single and multiple clients both.
> 
> If I test it in a distributed-multithreaded way,I do not see much of a boost.
> 
> On a single client,though it's a different story..I see almost 200% increase
> on seq writes (RHEL 7 servers,RHEL 6 clients,8 threads,1 client,64k rec
> size).
> 

What I'm seeing in comment #11 on my setup is that with client-io-threads off, you can still hit a throughput of about 500MB/s (YMMV). But the client can potentially deliver close to 800MB/s before it hits network saturation (since this is 10GbE and there is a 1.5x overhead from EC 4+2). If I try to achieve this potential with client-io-threads off, by increasing number of threads pumping out IOs, I'm not able to, because of the fuse thread bottleneck. But with client-io-threads on, I'm able to go pretty much all the way to peak potential.

On a distributed iozone test with 6 servers, the max aggregate throughput that the servers can deliver is, from what I've seen, approx. 4GB/s before the disk saturates.
If you're doing the test with 4 clients, each client can potentially hit its max of about 800 MB/s without saturating the server. In this case, I'd expect client-io-threads-on to give a big boost over client-io-threads-off, for the same reason as in comment #11, provided you have a large enough number of threads per client.
If you're doing the test with 6 clients, each client can only get to about 650MB/s before the servers start to saturate. Will the fuse thread bottleneck kick in at that throughput level? On my setup, yes; and I do see a big boost from client-io-threads-on, when I have a large number of threads per client. But it is harder to predict what you might see on your setup.
If you're doing the test with 12 clients, each client can only get to <350MB/s before the servers saturate. In that case, there is a good chance client-io-threads on or off might not make any difference.

Comment 18 Ambarish 2016-10-25 08:36:20 UTC
This is on a 12*(4+2), across 6 servers and clients , mounted via FUSE: 

*On 6 clients,distributed-multithreaded iozone tests(kB/sec)* :

4 threads/client  : io-threads off - 2892380 ; io-threads on - 2919754 (+1%)
8 threads/client  : io-threads off - 1334734 ; io-threads on - 3796965 (+184%)
16 threads/client : io-threads off - 1220451 ; io-threads on - 4243807 (+247%)


*On a single client,multithreaded iozone tests(kB/sec)* :

4 threads/client  : io-threads off - 474285  ; io-threads on - 477941 (+1%)
8 threads/client  : io-threads off - 211863  ; io-threads on - 684977 (+223%)
16 threads/client : io-threads off - 204642  ; io-threads on - 742063 (+262%)


I was able to hit the FUSE bottleneck,the magnitude of increase in write throughputs is higher,with more number of threads/client.

Comment 20 errata-xmlrpc 2017-03-23 05:39:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.