Bug 1499649

Description of problem:

The option "performance.client-io-threads on" is useful in situations where the fuse-thread bottlenecks. We see such situations often with disperse volumes> But we also see it with other volume types, esp. on small random I/O.

The number of client io-threads started is controlled by the option performance.io-thread-count, which by default is 16. That number is generally too high for client-io-threads. However, reducing io-thread-count can introduce bottlenecks on the brick side, since the same option controls number of io-threads on both client-side and brick. 

Ideally, we would have a separate option for controlling the number of client-side io-threads.

Version-Release number of selected component (if applicable):
glusterfs*-3.12.1-2.el7.x86_64


Actual results:

I have a single-brick distribute volume created on ramdisk. On this I run a random I/O read workload with 4k block size.

Results with different options:

performance.client-io-threads off: 63.9MiB/s i.e approx. 16k iops
[in the above, the fuse thread is the bottleneck at 99% CPU utilization]


[impact of turning client-io-threads on, without tuning io-thread-count, which by default is 16. mutrace seemed to report contention among the client-io-threads in this case; need to dig more into that]
performance.client-io-threads on : 98.5MiB/s

performance.client-io-threads on,
performance.io-thread-count 2    : 104MiB/s

performance.client-io-threads on,
performance.io-thread-count 1    : 69.7MiB/s

In the run with io-thread-count=1, there seems to be a bottleneck at the brick:
  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                    
 9263 root      20   0  682268  20792   3872 R 99.9  0.0   1:00.43 glusteriotwr0                              
 9166 root      20   0 1248736  28304   4956 R 50.0  0.1   0:29.33 glusteriotwr0 

Ideally, we would like to set number of client-io-threads to 1 or 2: just enough to eliminate the fuse-thread bottleneck, but not so high that it introduces contention among the client-io-threads. But depending on the nature of the brick device we may not want to reduce io-thread-count.


Additional info:

Thanks to Ravi, for clarifying io-threads behavior and providing the pointer:
https://github.com/gluster/glusterfs/commit/09232fd6855f288c47b5396dcd4d4245a154576f

Krutika also reports that a high number of client-io-threads is not needed: https://bugzilla.redhat.com/show_bug.cgi?id=1467614#c16. Some of the other observations in this bz differs from hers though.