Bug 1417815

Summary: [RFE] Support multiple bricks in one process (multiplexing)
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Atin Mukherjee <amukherj>
Component: coreAssignee: Samikshan Bairagya <sbairagy>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.2CC: asriram, asrivast, bugs, jdarcy, nchilaka, rhs-bugs, sbairagy, storage-qa-internal
Target Milestone: ---Keywords: FutureFeature, Triaged
Target Release: RHGS 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8.4-19 Doc Type: Enhancement
Doc Text:
Brick multiplexing can now be enabled for container workloads. This allows compatible bricks on the same node to use a single process and port, reducing port and process consumption.
Story Points: ---
Clone Of: 1385758
: 1436697 (view as bug list) Environment:
Last Closed: 2017-09-21 04:30:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1385758, 1418095, 1420434    
Bug Blocks: 1417138, 1436697    

Description Atin Mukherjee 2017-01-31 07:28:56 UTC
+++ This bug was initially created as a clone of Bug #1385758 +++

Primary goal is to support more bricks/volumes by reducing port and memory consumption.  Secondarily, this could improve performance in the long term - though so far performance is degraded slightly.  Feature page is here:

https://github.com/gluster/glusterfs-specs/blob/master/under_review/multiplexing.md

(note that this might move to a different directory as the feature moves through the upstream feature approval/tracking process)

--- Additional comment from Worker Ant on 2016-10-17 13:12:37 EDT ---

REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#4) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-17 19:40:49 EDT ---

REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#5) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-17 20:00:00 EDT ---

REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#6) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-24 15:47:39 EDT ---

REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#4) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-26 12:57:32 EDT ---

REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#5) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-26 19:38:21 EDT ---

REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#6) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-27 08:31:13 EDT ---

REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#7) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-27 12:19:28 EDT ---

REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#8) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-27 22:09:00 EDT ---

REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#7) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-10-28 15:31:56 EDT ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#22) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-11-07 11:57:58 EST ---

REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#8) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-11-07 12:28:12 EST ---

REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#9) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-11-17 17:16:04 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#23) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-11-23 17:41:19 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#24) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-11-23 19:44:49 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#25) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-01 15:25:27 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#26) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-02 16:38:35 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#27) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-08 16:27:10 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#28) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-12 14:08:54 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#29) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-12 21:50:47 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#30) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-12 21:55:29 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#31) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-12 23:03:30 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#32) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-13 08:08:05 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#33) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-14 12:29:29 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#34) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-14 12:33:28 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#35) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-14 12:55:02 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#36) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-14 13:59:38 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#37) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-14 16:42:07 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#38) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-14 22:00:02 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#39) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-16 16:30:10 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#40) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-20 15:51:13 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#41) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2016-12-21 13:16:24 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#42) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-09 11:19:47 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#43) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-09 11:19:56 EST ---

REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#10) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-10 17:28:11 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#44) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-11 13:11:52 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#45) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-12 13:30:55 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#46) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-12 14:15:31 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#5) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-12 17:17:09 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#6) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-12 19:27:50 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#7) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-12 23:14:16 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#8) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-12 23:43:10 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#47) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-13 13:52:23 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#48) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-13 13:53:13 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#9) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-17 11:21:43 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#49) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-17 11:22:04 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#10) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-17 17:42:14 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#50) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-17 17:44:26 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#11) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-18 09:04:55 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#51) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-18 09:05:11 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#12) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-18 12:30:20 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#52) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-18 14:29:32 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#13) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-19 16:33:18 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#53) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-19 16:50:50 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#14) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-19 20:29:03 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#15) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-20 12:47:43 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#54) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-20 15:23:01 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#55) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-20 17:33:40 EST ---

REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#56) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-20 17:34:09 EST ---

REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#16) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-24 07:29:58 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#59) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-24 12:48:14 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#60) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-24 22:24:45 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#61) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-24 23:46:08 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#62) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-25 08:54:06 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#63) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-25 09:34:42 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#64) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-25 10:19:16 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#65) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-25 13:25:02 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#66) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-25 21:27:52 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#67) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-26 11:15:40 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#68) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-26 12:44:18 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#69) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-26 13:05:05 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#70) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-26 15:18:30 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#71) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-26 15:25:58 EST ---

REVIEW: https://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#18) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-26 16:32:53 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#72) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-26 16:33:16 EST ---

REVIEW: https://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#19) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-27 17:43:36 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#73) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-27 20:04:07 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#74) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-28 07:16:56 EST ---

REVIEW: https://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#11) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-30 13:18:08 EST ---

REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#75) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-30 14:30:52 EST ---

REVIEW: https://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#12) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Worker Ant on 2017-01-30 19:14:02 EST ---

COMMIT: https://review.gluster.org/14763 committed in master by Vijay Bellur (vbellur) 
------
commit 1a95fc3036db51b82b6a80952f0908bc2019d24a
Author: Jeff Darcy <jdarcy>
Date:   Thu Dec 8 16:24:15 2016 -0500

    core: run many bricks within one glusterfsd process
    
    This patch adds support for multiple brick translator stacks running
    in a single brick server process.  This reduces our per-brick memory usage by
    approximately 3x, and our appetite for TCP ports even more.  It also creates
    potential to avoid process/thread thrashing, and to improve QoS by scheduling
    more carefully across the bricks, but realizing that potential will require
    further work.
    
    Multiplexing is controlled by the "cluster.brick-multiplex" global option.  By
    default it's off, and bricks are started in separate processes as before.  If
    multiplexing is enabled, then *compatible* bricks (mostly those with the same
    transport options) will be started in the same process.
    
    Change-Id: I45059454e51d6f4cbb29a4953359c09a408695cb
    BUG: 1385758
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/14763
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 5 Nag Pavan Chilakam 2017-04-24 06:53:51 UTC
QA validation:
Brick multiplexing's main and core theme of enabling multiple bricks to work on a single process, is working.
At a sanity level this RFE is working as expected and hence moving this bug to verified as there are no SHIP_STOPPERS OR QE_TEST_BLOCKERS at this point. 
however, that doesn't mean the testing is complete. QE will be still be testing this feature and will raise any BZs hit.

However, note that there are bugs reported due to this change and can be seen with the below query:

https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&bug_status=MODIFIED&bug_status=ON_DEV&bug_status=ON_QA&bug_status=VERIFIED&bug_status=RELEASE_PENDING&bug_status=CLOSED&f1=cf_qa_whiteboard&known_name=brickmultiplexing%20bugs&list_id=7285160&o1=anywordssubstr&query_based_on=brickmultiplexing%20bugs&query_format=advanced&v1=brick_multiplexing

Also the test status can be tracked at https://docs.google.com/spreadsheets/d/1Evx_zbxflQUWi4axF2oGPFqnsYqgxXwvfKW4_fNfDc0/edit#gid=75413810

test versions:3.8.4-22

Comment 9 errata-xmlrpc 2017-09-21 04:30:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774

Comment 10 errata-xmlrpc 2017-09-21 04:56:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774