Bug 1419305 - disable client.io-threads on replica volume creation
Summary: disable client.io-threads on replica volume creation
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: replicate
Version: 3.10
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Atin Mukherjee
QA Contact:
URL:
Whiteboard:
Depends On: 1418014
Blocks: 1418011
TreeView+ depends on / blocked
 
Reported: 2017-02-05 07:03 UTC by Atin Mukherjee
Modified: 2017-03-06 17:45 UTC (History)
5 users (show)

Fixed In Version: glusterfs-3.10.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1418014
Environment:
Last Closed: 2017-03-06 17:45:31 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Atin Mukherjee 2017-02-05 07:03:41 UTC
+++ This bug was initially created as a clone of Bug #1418014 +++

+++ This bug was initially created as a clone of Bug #1418011 +++

Description of problem:

client.io-threads is turned on by default in rhgs-3.2.0. While this tunable has improved EC performance significantly, it has adverse effects on the replicate volume performance, following BZs were filed by perf QE while validating this option:


> https://bugzilla.redhat.com/show_bug.cgi?id=1413512 : [Perf]  Inconsistent sequential writes on FUSE due to client-io-threads

> https://bugzilla.redhat.com/show_bug.cgi?id=1404113 : [Perf] 12% Drop in  sequential reads on SMB v1.0

> https://bugzilla.redhat.com/show_bug.cgi?id=1397854 : [Perf] 10% and 20% drop in sequential writes  on SMB v1 and V3 with RHEL 6.8

> https://bugzilla.redhat.com/show_bug.cgi?id=1395204 : 34% drop in Random Writes  from 3.1.3 to 3.2 on FUSE


As these issues can not be addressed within rhgs-3.2.0 timelines as they are related to the design limitation of AFR, it was decided to turn this option off when a replicate volume is created.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-01-31 10:30:36 EST ---

This bug is automatically being proposed for the current release of Red Hat Gluster Storage 3 under active development, by setting the release flag 'rhgs‑3.2.0' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Atin Mukherjee on 2017-01-31 10:35:46 EST ---

Description of problem:

client.io-threads is turned on by default in 3.9 release. While this tunable has improved EC performance significantly, it has adverse effects on the replicate volume performance, following BZs were filed by perf QE while validating this option:


> https://bugzilla.redhat.com/show_bug.cgi?id=1413512 : [Perf]  Inconsistent sequential writes on FUSE due to client-io-threads

> https://bugzilla.redhat.com/show_bug.cgi?id=1404113 : [Perf] 12% Drop in  sequential reads on SMB v1.0

> https://bugzilla.redhat.com/show_bug.cgi?id=1397854 : [Perf] 10% and 20% drop in sequential writes  on SMB v1 and V3 with RHEL 6.8

> https://bugzilla.redhat.com/show_bug.cgi?id=1395204 : 34% drop in Random Writes  from 3.1.3 to 3.2 on FUSE


As these issues can not be addressed immediately as they are related to the design limitation of AFR, we should turn this option off when a replicate volume is created as a preventive measure.

--- Additional comment from Worker Ant on 2017-01-31 10:44:40 EST ---

REVIEW: https://review.gluster.org/16492 (glusterd: turn of client.io-threads for replicate volumes) posted (#2) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Worker Ant on 2017-01-31 12:13:10 EST ---

REVIEW: https://review.gluster.org/16492 (glusterd: turn of client.io-threads for replicate volumes) posted (#3) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Worker Ant on 2017-01-31 23:34:26 EST ---

REVIEW: https://review.gluster.org/16492 (glusterd: turn off client.io-threads for replicate volumes) posted (#4) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Worker Ant on 2017-02-01 01:12:13 EST ---

REVIEW: https://review.gluster.org/16492 (glusterd: turn off client.io-threads for replicate volumes) posted (#5) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Worker Ant on 2017-02-01 01:15:45 EST ---

REVIEW: https://review.gluster.org/16492 (glusterd: turn off client.io-threads for replicate volumes) posted (#6) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Worker Ant on 2017-02-01 09:56:52 EST ---

REVIEW: https://review.gluster.org/16502 (glusterd : do not load io-threads in client graph for replicate volumes) posted (#5) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Worker Ant on 2017-02-03 07:54:53 EST ---

COMMIT: https://review.gluster.org/16502 committed in master by Jeff Darcy (jdarcy@redhat.com) 
------
commit ff075a3d6f9b142911d25c27fd209838782bfff0
Author: Atin Mukherjee <amukherj@redhat.com>
Date:   Wed Feb 1 15:04:08 2017 +0530

    glusterd : do not load io-threads in client graph for replicate volumes
    
    client.io-threads has been turned on by default from release-3.9 onwards,
    however this has an adverse effects on replicate volumes due to the design
    limitations on replications, till that gets addressed through server side
    replication as a preventive measure it is wiser not to load io-threads in the
    client graph for replicate volumes.
    
    Change-Id: Ibc576d4517da23fcdf55c6f4d17b90152a8817d7
    BUG: 1418014
    Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
    Reviewed-on: https://review.gluster.org/16502
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Prashanth Pai <ppai@redhat.com>
    Reviewed-by: Ravishankar N <ravishankar@redhat.com>
    Reviewed-by: Jeff Darcy <jdarcy@redhat.com>

Comment 1 Worker Ant 2017-02-05 07:05:12 UTC
REVIEW: https://review.gluster.org/16545 (glusterd : do not load io-threads in client graph for replicate volumes) posted (#1) for review on release-3.10 by Atin Mukherjee (amukherj@redhat.com)

Comment 2 Worker Ant 2017-02-06 14:53:37 UTC
COMMIT: https://review.gluster.org/16545 committed in release-3.10 by Shyamsundar Ranganathan (srangana@redhat.com) 
------
commit 4c919ed5311dac4d135b5f9d1e682a9b3ef8ffbb
Author: Atin Mukherjee <amukherj@redhat.com>
Date:   Wed Feb 1 15:04:08 2017 +0530

    glusterd : do not load io-threads in client graph for replicate volumes
    
    client.io-threads has been turned on by default from release-3.9 onwards,
    however this has an adverse effects on replicate volumes due to the design
    limitations on replications, till that gets addressed through server side
    replication as a preventive measure it is wiser not to load io-threads in the
    client graph for replicate volumes.
    
    >Reviewed-on: https://review.gluster.org/16502
    >NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    >CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    >Smoke: Gluster Build System <jenkins@build.gluster.org>
    >Reviewed-by: Prashanth Pai <ppai@redhat.com>
    >Reviewed-by: Ravishankar N <ravishankar@redhat.com>
    >Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
    
    Change-Id: Ibc576d4517da23fcdf55c6f4d17b90152a8817d7
    BUG: 1419305
    Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
    Reviewed-on: https://review.gluster.org/16545
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Prashanth Pai <ppai@redhat.com>
    Reviewed-by: Ravishankar N <ravishankar@redhat.com>

Comment 3 Shyamsundar 2017-03-06 17:45:31 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.