+++ This bug was initially created as a clone of Bug #1385758 +++ Primary goal is to support more bricks/volumes by reducing port and memory consumption. Secondarily, this could improve performance in the long term - though so far performance is degraded slightly. Feature page is here: https://github.com/gluster/glusterfs-specs/blob/master/under_review/multiplexing.md (note that this might move to a different directory as the feature moves through the upstream feature approval/tracking process) --- Additional comment from Worker Ant on 2016-10-17 13:12:37 EDT --- REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#4) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-17 19:40:49 EDT --- REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#5) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-17 20:00:00 EDT --- REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#6) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-24 15:47:39 EDT --- REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#4) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-26 12:57:32 EDT --- REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#5) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-26 19:38:21 EDT --- REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#6) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-27 08:31:13 EDT --- REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#7) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-27 12:19:28 EDT --- REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#8) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-27 22:09:00 EDT --- REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#7) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-10-28 15:31:56 EDT --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#22) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-11-07 11:57:58 EST --- REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#8) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-11-07 12:28:12 EST --- REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#9) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-11-17 17:16:04 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#23) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-11-23 17:41:19 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#24) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-11-23 19:44:49 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#25) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-01 15:25:27 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#26) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-02 16:38:35 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#27) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-08 16:27:10 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#28) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-12 14:08:54 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#29) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-12 21:50:47 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#30) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-12 21:55:29 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#31) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-12 23:03:30 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#32) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-13 08:08:05 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#33) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-14 12:29:29 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#34) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-14 12:33:28 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#35) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-14 12:55:02 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#36) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-14 13:59:38 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#37) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-14 16:42:07 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#38) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-14 22:00:02 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#39) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-16 16:30:10 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#40) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-20 15:51:13 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#41) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2016-12-21 13:16:24 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#42) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-09 11:19:47 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#43) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-09 11:19:56 EST --- REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#10) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-10 17:28:11 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#44) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-11 13:11:52 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#45) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-12 13:30:55 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#46) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-12 14:15:31 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#5) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-12 17:17:09 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#6) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-12 19:27:50 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#7) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-12 23:14:16 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#8) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-12 23:43:10 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#47) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-13 13:52:23 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#48) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-13 13:53:13 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#9) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-17 11:21:43 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#49) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-17 11:22:04 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#10) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-17 17:42:14 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#50) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-17 17:44:26 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#11) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-18 09:04:55 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#51) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-18 09:05:11 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#12) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-18 12:30:20 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#52) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-18 14:29:32 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#13) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-19 16:33:18 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#53) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-19 16:50:50 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#14) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-19 20:29:03 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#15) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-20 12:47:43 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#54) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-20 15:23:01 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#55) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-20 17:33:40 EST --- REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#56) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-20 17:34:09 EST --- REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#16) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-24 07:29:58 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#59) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-24 12:48:14 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#60) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-24 22:24:45 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#61) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-24 23:46:08 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#62) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-25 08:54:06 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#63) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-25 09:34:42 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#64) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-25 10:19:16 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#65) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-25 13:25:02 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#66) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-25 21:27:52 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#67) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-26 11:15:40 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#68) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-26 12:44:18 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#69) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-26 13:05:05 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#70) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-26 15:18:30 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#71) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-26 15:25:58 EST --- REVIEW: https://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#18) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-26 16:32:53 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#72) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-26 16:33:16 EST --- REVIEW: https://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#19) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-27 17:43:36 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#73) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-27 20:04:07 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#74) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-28 07:16:56 EST --- REVIEW: https://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#11) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-30 13:18:08 EST --- REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#75) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-30 14:30:52 EST --- REVIEW: https://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#12) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Worker Ant on 2017-01-30 19:14:02 EST --- COMMIT: https://review.gluster.org/14763 committed in master by Vijay Bellur (vbellur) ------ commit 1a95fc3036db51b82b6a80952f0908bc2019d24a Author: Jeff Darcy <jdarcy> Date: Thu Dec 8 16:24:15 2016 -0500 core: run many bricks within one glusterfsd process This patch adds support for multiple brick translator stacks running in a single brick server process. This reduces our per-brick memory usage by approximately 3x, and our appetite for TCP ports even more. It also creates potential to avoid process/thread thrashing, and to improve QoS by scheduling more carefully across the bricks, but realizing that potential will require further work. Multiplexing is controlled by the "cluster.brick-multiplex" global option. By default it's off, and bricks are started in separate processes as before. If multiplexing is enabled, then *compatible* bricks (mostly those with the same transport options) will be started in the same process. Change-Id: I45059454e51d6f4cbb29a4953359c09a408695cb BUG: 1385758 Signed-off-by: Jeff Darcy <jdarcy> Reviewed-on: https://review.gluster.org/14763 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Vijay Bellur <vbellur>
downstream patches: https://code.engineering.redhat.com/gerrit/#/c/101300/ https://code.engineering.redhat.com/gerrit/#/c/101301/ https://code.engineering.redhat.com/gerrit/#/c/101302/ https://code.engineering.redhat.com/gerrit/#/c/101303/ https://code.engineering.redhat.com/gerrit/#/c/101304 https://code.engineering.redhat.com/gerrit/#/c/101305/ https://code.engineering.redhat.com/gerrit/#/c/101308/ https://code.engineering.redhat.com/gerrit/#/c/101309/ https://code.engineering.redhat.com/gerrit/#/c/101313/ https://code.engineering.redhat.com/gerrit/#/c/101315/ https://code.engineering.redhat.com/gerrit/#/c/101317 https://code.engineering.redhat.com/gerrit/#/c/101318/ https://code.engineering.redhat.com/gerrit/#/c/101322/ https://code.engineering.redhat.com/gerrit/#/c/101326/ https://code.engineering.redhat.com/gerrit/#/c/101327/ https://code.engineering.redhat.com/gerrit/#/c/101328
QA validation: Brick multiplexing's main and core theme of enabling multiple bricks to work on a single process, is working. At a sanity level this RFE is working as expected and hence moving this bug to verified as there are no SHIP_STOPPERS OR QE_TEST_BLOCKERS at this point. however, that doesn't mean the testing is complete. QE will be still be testing this feature and will raise any BZs hit. However, note that there are bugs reported due to this change and can be seen with the below query: https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&bug_status=MODIFIED&bug_status=ON_DEV&bug_status=ON_QA&bug_status=VERIFIED&bug_status=RELEASE_PENDING&bug_status=CLOSED&f1=cf_qa_whiteboard&known_name=brickmultiplexing%20bugs&list_id=7285160&o1=anywordssubstr&query_based_on=brickmultiplexing%20bugs&query_format=advanced&v1=brick_multiplexing Also the test status can be tracked at https://docs.google.com/spreadsheets/d/1Evx_zbxflQUWi4axF2oGPFqnsYqgxXwvfKW4_fNfDc0/edit#gid=75413810 test versions:3.8.4-22
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774