Description of problem: Turning on client.io-threads improves performance for parallel I/O from single mount point on EC/dist-rep volumes. This request is to turn on the option by default. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Upstream patch : http://review.gluster.org/#/c/15051/ is merged in mainline.
Downstream patch: https://code.engineering.redhat.com/gerrit/#/c/84684/
Downstream patch is merged now.
Verified this using the build - glusterfs-3.8.4-2 i am seeing volume option performace.client-io-threads enabled by default. The default value of performace.client-io-threads in 3.1.3 and 3.2 ( 3.8.4-2 ) is below. In 3.1.3 ======== ]# gluster volume get Dis performance.client-io-threads Option Value ------ ----- performance.client-io-threads off [root@ ~]# In 3.2 ====== ~]# gluster volume get Replica performance.client-io-threads Option Value ------ ----- performance.client-io-threads on [root@ ~]# Based on above details moving to verified state.
Quantifying the performance benefit of enabling client-io-threads on a 12x(4+2) volume (on 6 servers, with 12 drives per server in JBOD mode) with the following options set: cluster.lookup-optimize: on server.event-threads: 4 client.event-threads: 4 Test: <body> MY_MNT="/mnt/glustervol" for ii in 1 4 8; do iozone -i 0 -w -+n -c -e -s 10g -r 64k -t ${ii} -F ${MY_MNT}/f{1..8}.ioz rm -f ${MY_MNT}/f*.ioz done </body> Result: No. of threads | client-io-threads=off | client-io-threads=on 1 | 136125.80 kB/sec | 140190.06 kB/sec 4 | 505460.07 kB/sec | 495928.16 kB/sec 8 | 177709.80 kB/sec | 785245.68 kB/sec
Ambarish, could you post a followup to comment #11 with results that you see on your setup for the same test? It will help with some of the discussions that are going on. Note: these are single-client iozone tests, not distributed.
Manoj, While testing this on 3.1.3,I tested using single and multiple clients both. If I test it in a distributed-multithreaded way,I do not see much of a boost. On a single client,though it's a different story..I see almost 200% increase on seq writes (RHEL 7 servers,RHEL 6 clients,8 threads,1 client,64k rec size). Are the results on a single client good enough to make this param "on" by default in production? If yes,we might need to advertise/communicate the exact config/scenarios where we are seeing major perf boosts.
(In reply to Ambarish from comment #13) > Manoj, > > While testing this on 3.1.3,I tested using single and multiple clients both. > > If I test it in a distributed-multithreaded way,I do not see much of a boost. > > On a single client,though it's a different story..I see almost 200% increase > on seq writes (RHEL 7 servers,RHEL 6 clients,8 threads,1 client,64k rec > size). > > Are the results on a single client good enough to make this param "on" by > default in production? Yes, I think so. These are all approximations of workloads customers might see in production. > > If yes,we might need to advertise/communicate the exact config/scenarios > where we are seeing major perf boosts. The actual flow of requests for EC is somewhat complex from what I understand. Which thread(s) end up doing the heavy lifting aka encoding for writes depends on dynamic conditions -- sometimes it can be the fuse-reader thread (or client-io-threads, if enabled), sometimes it can be the client.event-threads. I think eventually we want to be able to give users a simple guideline: turn on client-io-threads if you're using EC. Or better still, make client-io-threads=on the default, as this bz does (but, from what I understand, we are going to reverse this for RHGS 3.2 because of bugs we have hit).
(In reply to Manoj Pillai from comment #15) > (In reply to Ambarish from comment #13) > > Manoj, > > > > While testing this on 3.1.3,I tested using single and multiple clients both. > > > > If I test it in a distributed-multithreaded way,I do not see much of a boost. > > > > On a single client,though it's a different story..I see almost 200% increase > > on seq writes (RHEL 7 servers,RHEL 6 clients,8 threads,1 client,64k rec > > size). > > > > Are the results on a single client good enough to make this param "on" by > > default in production? > > Yes, I think so. These are all approximations of workloads customers might > see in production. > > > > > If yes,we might need to advertise/communicate the exact config/scenarios > > where we are seeing major perf boosts. > > The actual flow of requests for EC is somewhat complex from what I > understand. Which thread(s) end up doing the heavy lifting aka encoding for > writes depends on dynamic conditions -- sometimes it can be the fuse-reader > thread (or client-io-threads, if enabled), sometimes it can be the > client.event-threads. > > I think eventually we want to be able to give users a simple guideline: turn > on client-io-threads if you're using EC. Or better still, make > client-io-threads=on the default, as this bz does (but, from what I > understand, we are going to reverse this for RHGS 3.2 because of bugs we > have hit). Nope, it is better to fix them :-)
(In reply to Ambarish from comment #13) > Manoj, > > While testing this on 3.1.3,I tested using single and multiple clients both. > > If I test it in a distributed-multithreaded way,I do not see much of a boost. > > On a single client,though it's a different story..I see almost 200% increase > on seq writes (RHEL 7 servers,RHEL 6 clients,8 threads,1 client,64k rec > size). > What I'm seeing in comment #11 on my setup is that with client-io-threads off, you can still hit a throughput of about 500MB/s (YMMV). But the client can potentially deliver close to 800MB/s before it hits network saturation (since this is 10GbE and there is a 1.5x overhead from EC 4+2). If I try to achieve this potential with client-io-threads off, by increasing number of threads pumping out IOs, I'm not able to, because of the fuse thread bottleneck. But with client-io-threads on, I'm able to go pretty much all the way to peak potential. On a distributed iozone test with 6 servers, the max aggregate throughput that the servers can deliver is, from what I've seen, approx. 4GB/s before the disk saturates. If you're doing the test with 4 clients, each client can potentially hit its max of about 800 MB/s without saturating the server. In this case, I'd expect client-io-threads-on to give a big boost over client-io-threads-off, for the same reason as in comment #11, provided you have a large enough number of threads per client. If you're doing the test with 6 clients, each client can only get to about 650MB/s before the servers start to saturate. Will the fuse thread bottleneck kick in at that throughput level? On my setup, yes; and I do see a big boost from client-io-threads-on, when I have a large number of threads per client. But it is harder to predict what you might see on your setup. If you're doing the test with 12 clients, each client can only get to <350MB/s before the servers saturate. In that case, there is a good chance client-io-threads on or off might not make any difference.
This is on a 12*(4+2), across 6 servers and clients , mounted via FUSE: *On 6 clients,distributed-multithreaded iozone tests(kB/sec)* : 4 threads/client : io-threads off - 2892380 ; io-threads on - 2919754 (+1%) 8 threads/client : io-threads off - 1334734 ; io-threads on - 3796965 (+184%) 16 threads/client : io-threads off - 1220451 ; io-threads on - 4243807 (+247%) *On a single client,multithreaded iozone tests(kB/sec)* : 4 threads/client : io-threads off - 474285 ; io-threads on - 477941 (+1%) 8 threads/client : io-threads off - 211863 ; io-threads on - 684977 (+223%) 16 threads/client : io-threads off - 204642 ; io-threads on - 742063 (+262%) I was able to hit the FUSE bottleneck,the magnitude of increase in write throughputs is higher,with more number of threads/client.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html