Bug 1499649 - Need option to control number of client-io-threads [NEEDINFO]
Summary: Need option to control number of client-io-threads
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: GlusterFS
Classification: Community
Component: io-threads
Version: mainline
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Pranith Kumar K
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-09 08:10 UTC by Manoj Pillai
Modified: 2018-11-20 09:44 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-20 09:11:00 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
srangana: needinfo? (pkarampu)


Attachments (Terms of Use)

Description Manoj Pillai 2017-10-09 08:10:37 UTC
Description of problem:

The option "performance.client-io-threads on" is useful in situations where the fuse-thread bottlenecks. We see such situations often with disperse volumes> But we also see it with other volume types, esp. on small random I/O.

The number of client io-threads started is controlled by the option performance.io-thread-count, which by default is 16. That number is generally too high for client-io-threads. However, reducing io-thread-count can introduce bottlenecks on the brick side, since the same option controls number of io-threads on both client-side and brick. 

Ideally, we would have a separate option for controlling the number of client-side io-threads.

Version-Release number of selected component (if applicable):
glusterfs*-3.12.1-2.el7.x86_64


Actual results:

I have a single-brick distribute volume created on ramdisk. On this I run a random I/O read workload with 4k block size.

Results with different options:

performance.client-io-threads off: 63.9MiB/s i.e approx. 16k iops
[in the above, the fuse thread is the bottleneck at 99% CPU utilization]


[impact of turning client-io-threads on, without tuning io-thread-count, which by default is 16. mutrace seemed to report contention among the client-io-threads in this case; need to dig more into that]
performance.client-io-threads on : 98.5MiB/s

performance.client-io-threads on,
performance.io-thread-count 2    : 104MiB/s

performance.client-io-threads on,
performance.io-thread-count 1    : 69.7MiB/s

In the run with io-thread-count=1, there seems to be a bottleneck at the brick:
  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                    
 9263 root      20   0  682268  20792   3872 R 99.9  0.0   1:00.43 glusteriotwr0                              
 9166 root      20   0 1248736  28304   4956 R 50.0  0.1   0:29.33 glusteriotwr0 

Ideally, we would like to set number of client-io-threads to 1 or 2: just enough to eliminate the fuse-thread bottleneck, but not so high that it introduces contention among the client-io-threads. But depending on the nature of the brick device we may not want to reduce io-thread-count.


Additional info:

Thanks to Ravi, for clarifying io-threads behavior and providing the pointer:
https://github.com/gluster/glusterfs/commit/09232fd6855f288c47b5396dcd4d4245a154576f

Krutika also reports that a high number of client-io-threads is not needed: https://bugzilla.redhat.com/show_bug.cgi?id=1467614#c16. Some of the other observations in this bz differs from hers though.

Comment 1 Manoj Pillai 2017-10-09 08:43:42 UTC
(In reply to Manoj Pillai from comment #0)
[...]
> 
> In the run with io-thread-count=1, there seems to be a bottleneck at the
> brick:
>   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND  
> 
>  9263 root      20   0  682268  20792   3872 R 99.9  0.0   1:00.43
> glusteriotwr0                              
>  9166 root      20   0 1248736  28304   4956 R 50.0  0.1   0:29.33
> glusteriotwr0 
> 

Actually, now I'm not positive that the bottleneck is on the brick-side. Need to re-check.

But the logic of this bz is still valid, IMO. :).

Comment 2 Manoj Pillai 2017-10-09 17:17:27 UTC
(In reply to Manoj Pillai from comment #0)
[...]
> 
> Ideally, we would like to set number of client-io-threads to 1 or 2: 

The above does not apply to disperse, where you might need a higher number of client-io-threads because of fairly cpu-intensive encoding/decoding tasks that can come to client-io-threads.

Comment 3 Shyamsundar 2018-10-23 14:53:54 UTC
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions.

Comment 4 Yaniv Kaul 2018-11-01 13:16:08 UTC
(In reply to Shyamsundar from comment #3)
> Release 3.12 has been EOLd and this bug was still found to be in the NEW
> state, hence moving the version to mainline, to triage the same and take
> appropriate actions.

When will it be triaged?

Comment 5 Shyamsundar 2018-11-01 14:08:30 UTC
Marked assignment and changed status of *this* carry over bug.

Comment 6 Vijay Bellur 2018-11-20 09:44:51 UTC
Migrated to github:

https://github.com/gluster/glusterfs/issues/607

Please follow the github issue for further updates on this bug.


Note You need to log in before you can comment on or make changes to this bug.