Description of problem: The option "performance.client-io-threads on" is useful in situations where the fuse-thread bottlenecks. We see such situations often with disperse volumes> But we also see it with other volume types, esp. on small random I/O. The number of client io-threads started is controlled by the option performance.io-thread-count, which by default is 16. That number is generally too high for client-io-threads. However, reducing io-thread-count can introduce bottlenecks on the brick side, since the same option controls number of io-threads on both client-side and brick. Ideally, we would have a separate option for controlling the number of client-side io-threads. Version-Release number of selected component (if applicable): glusterfs*-3.12.1-2.el7.x86_64 Actual results: I have a single-brick distribute volume created on ramdisk. On this I run a random I/O read workload with 4k block size. Results with different options: performance.client-io-threads off: 63.9MiB/s i.e approx. 16k iops [in the above, the fuse thread is the bottleneck at 99% CPU utilization] [impact of turning client-io-threads on, without tuning io-thread-count, which by default is 16. mutrace seemed to report contention among the client-io-threads in this case; need to dig more into that] performance.client-io-threads on : 98.5MiB/s performance.client-io-threads on, performance.io-thread-count 2 : 104MiB/s performance.client-io-threads on, performance.io-thread-count 1 : 69.7MiB/s In the run with io-thread-count=1, there seems to be a bottleneck at the brick: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9263 root 20 0 682268 20792 3872 R 99.9 0.0 1:00.43 glusteriotwr0 9166 root 20 0 1248736 28304 4956 R 50.0 0.1 0:29.33 glusteriotwr0 Ideally, we would like to set number of client-io-threads to 1 or 2: just enough to eliminate the fuse-thread bottleneck, but not so high that it introduces contention among the client-io-threads. But depending on the nature of the brick device we may not want to reduce io-thread-count. Additional info: Thanks to Ravi, for clarifying io-threads behavior and providing the pointer: https://github.com/gluster/glusterfs/commit/09232fd6855f288c47b5396dcd4d4245a154576f Krutika also reports that a high number of client-io-threads is not needed: https://bugzilla.redhat.com/show_bug.cgi?id=1467614#c16. Some of the other observations in this bz differs from hers though.
(In reply to Manoj Pillai from comment #0) [...] > > In the run with io-thread-count=1, there seems to be a bottleneck at the > brick: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 9263 root 20 0 682268 20792 3872 R 99.9 0.0 1:00.43 > glusteriotwr0 > 9166 root 20 0 1248736 28304 4956 R 50.0 0.1 0:29.33 > glusteriotwr0 > Actually, now I'm not positive that the bottleneck is on the brick-side. Need to re-check. But the logic of this bz is still valid, IMO. :).
(In reply to Manoj Pillai from comment #0) [...] > > Ideally, we would like to set number of client-io-threads to 1 or 2: The above does not apply to disperse, where you might need a higher number of client-io-threads because of fairly cpu-intensive encoding/decoding tasks that can come to client-io-threads.
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions.
(In reply to Shyamsundar from comment #3) > Release 3.12 has been EOLd and this bug was still found to be in the NEW > state, hence moving the version to mainline, to triage the same and take > appropriate actions. When will it be triaged?
Marked assignment and changed status of *this* carry over bug.
Migrated to github: https://github.com/gluster/glusterfs/issues/607 Please follow the github issue for further updates on this bug.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days