Bug 1487495
| Summary: | client-io-threads option not working for replicated volumes | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Manoj Pillai <mpillai> | |
| Component: | replicate | Assignee: | Ravishankar N <ravishankar> | |
| Status: | CLOSED ERRATA | QA Contact: | Vijay Avuthu <vavuthu> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | rhgs-3.3 | CC: | amukherj, ravishankar, rcyriac, rhinduja, rhs-bugs, sheggodu, storage-qa-internal | |
| Target Milestone: | --- | Keywords: | Performance | |
| Target Release: | RHGS 3.4.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | rebase | |||
| Fixed In Version: | glusterfs-3.12.2-1 | Doc Type: | If docs needed, set a value | |
| Doc Text: |
Previously, there was a bug in the code due to which client-io-threads was not loaded in a replicate volume, even though `gluster volume set <volname> client-io-threads on` returned success. With this fix, it works correctly.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1498570 1598416 (view as bug list) | Environment: | ||
| Last Closed: | 2018-09-04 06:35:11 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1498570, 1499158, 1503134 | |||
|
Description
Manoj Pillai
2017-09-01 05:44:34 UTC
Some data on the potential performance benefit from turning client-io-threads on ... fio random io test, with 16 threads, direct=1, block size=8. 1x3 volume with bricks on nvme. 10GbE network. fuse-mounted on client. no client-io-threads: read : io=8192.0MB, bw=113406K/s, iops=14175 write: io=8192.0MB, bw=65170K/s, iops=8146 [The fuse thread is at 99+% CPU utilization in the above tests.] client-io-threads turned on by manually editing volfiles [Thanks to Krutika for showing how to do that]: read : io=8192.0MB, bw=211604K/s, iops=26450 write: io=8192.0MB, bw=152025K/s, iops=19003 Read iops improves by 1.86x Write iops improves by 2.33x I understand that in perf QE testing we have seen performance degradations from turning client-io-threads on. No disputing that, but turning it on UNDER THE RIGHT CIRCUMSTANCES, i.e. when you are hitting a fuse thread bottleneck, can give huge benefits. We should be able to turn it on via the CLI in these scenarios. But based on perf QE results, we can keep it off by default. Upstream patch: https://review.gluster.org/18430 Ravi, my understanding is that the number of client-io-threads is controlled by the option performance.io-thread-count? And that this option also controls the number of io-threads at the brick. Can you confirm the above and that currently there is no way control the number of client-io-threads independently of the number of io-threads at the brick? That is correct. https://github.com/gluster/glusterfs/commit/09232fd6855f288c47b5396dcd4d4245a154576f seems to have introduced support for loading io-threads xlator on the client side but has not provided any specific way to modify the tunables exclusively for the brick or client side. So yes, changing any xlator option in iot xlator gets reflected in both the brick and client volfiles. Thanks! I opened an upstream bz for this along with some data in support of the request: https://bugzilla.redhat.com/show_bug.cgi?id=1499649. Update: ========= Build Used: glusterfs-3.12.2-8.el7rhgs.x86_64 Below Scenario's are verified: Scenario 1: 1) create 1 * 3 replicate volume and start it 2) check for performance.client-io-threads and it should be off 3) check in trusted-<volname>.tcp-fuse.vol file whether io-threads are loaded or not ( it shouldnot be loaded ) Scenario 2: 1) create distribute volume ( 1 * 1 ) and start 2) check for performance.client-io-threads and it should be on 3) io-threads should be loaded in trusted-<volname>.tcp-fuse.vol 4) add bricks to convert to replicate volume ( 1 * 2 ) 5) check for performance.client-io-threads and it should be off 6) io-threads shouldn't be loaded in trusted-<volname>.tcp-fuse.vol 7) remove bricks so that volume type will be converted to distribute 8) check for performance.client-io-threads and it should be on agian 9) io-threads should be loaded in trusted-<volname>.tcp-fuse.vol Scenario 3: 1) create 1 * 3 replicate volume and start it 2) set performance.client-io-threads to on 3) check for performance.client-io-threads and it should be on 4) io-threads should be loaded in trusted-<volname>.tcp-fuse.vol 5) add bricks so that volume converts to 2 * 3 6) check for performance.client-io-threads and it should be on 7) io-threads should be loaded in trusted-<volname>.tcp-fuse.vol 8) remove bricks so that 2 * 3 converts to 1 * 3 9) check for performance.client-io-threads and it should be on 10) io-threads should be loaded in trusted-<volname>.tcp-fuse.vol Changing status to Verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607 |