Bug 1385758 - [RFE] Support multiple bricks in one process (multiplexing)
Summary: [RFE] Support multiple bricks in one process (multiplexing)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: core
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jeff Darcy
QA Contact:
URL:
Whiteboard:
: 1360401 (view as bug list)
Depends On:
Blocks: 1417815 1418091 1436697
TreeView+ depends on / blocked
 
Reported: 2016-10-17 16:05 UTC by Jeff Darcy
Modified: 2017-05-30 18:35 UTC (History)
1 user (show)

Fixed In Version: glusterfs-3.11.0
Clone Of:
: 1417815 1418091 (view as bug list)
Environment:
Last Closed: 2017-03-09 20:33:39 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Jeff Darcy 2016-10-17 16:05:24 UTC
Primary goal is to support more bricks/volumes by reducing port and memory consumption.  Secondarily, this could improve performance in the long term - though so far performance is degraded slightly.  Feature page is here:

https://github.com/gluster/glusterfs-specs/blob/master/under_review/multiplexing.md

(note that this might move to a different directory as the feature moves through the upstream feature approval/tracking process)

Comment 1 Worker Ant 2016-10-17 17:12:37 UTC
REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#4) for review on master by Jeff Darcy (jdarcy)

Comment 2 Worker Ant 2016-10-17 23:40:49 UTC
REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#5) for review on master by Jeff Darcy (jdarcy)

Comment 3 Worker Ant 2016-10-18 00:00:00 UTC
REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#6) for review on master by Jeff Darcy (jdarcy)

Comment 4 Worker Ant 2016-10-24 19:47:39 UTC
REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#4) for review on master by Jeff Darcy (jdarcy)

Comment 5 Worker Ant 2016-10-26 16:57:32 UTC
REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#5) for review on master by Jeff Darcy (jdarcy)

Comment 6 Worker Ant 2016-10-26 23:38:21 UTC
REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#6) for review on master by Jeff Darcy (jdarcy)

Comment 7 Worker Ant 2016-10-27 12:31:13 UTC
REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#7) for review on master by Jeff Darcy (jdarcy)

Comment 8 Worker Ant 2016-10-27 16:19:28 UTC
REVIEW: http://review.gluster.org/15643 (io-threads: do global scaling instead of per-instance) posted (#8) for review on master by Jeff Darcy (jdarcy)

Comment 9 Worker Ant 2016-10-28 02:09:00 UTC
REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#7) for review on master by Jeff Darcy (jdarcy)

Comment 10 Worker Ant 2016-10-28 19:31:56 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#22) for review on master by Jeff Darcy (jdarcy)

Comment 11 Worker Ant 2016-11-07 16:57:58 UTC
REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#8) for review on master by Jeff Darcy (jdarcy)

Comment 12 Worker Ant 2016-11-07 17:28:12 UTC
REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#9) for review on master by Jeff Darcy (jdarcy)

Comment 13 Worker Ant 2016-11-17 22:16:04 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#23) for review on master by Jeff Darcy (jdarcy)

Comment 14 Worker Ant 2016-11-23 22:41:19 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#24) for review on master by Jeff Darcy (jdarcy)

Comment 15 Worker Ant 2016-11-24 00:44:49 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#25) for review on master by Jeff Darcy (jdarcy)

Comment 16 Worker Ant 2016-12-01 20:25:27 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#26) for review on master by Jeff Darcy (jdarcy)

Comment 17 Worker Ant 2016-12-02 21:38:35 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#27) for review on master by Jeff Darcy (jdarcy)

Comment 18 Worker Ant 2016-12-08 21:27:10 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#28) for review on master by Jeff Darcy (jdarcy)

Comment 19 Worker Ant 2016-12-12 19:08:54 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#29) for review on master by Jeff Darcy (jdarcy)

Comment 20 Worker Ant 2016-12-13 02:50:47 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#30) for review on master by Jeff Darcy (jdarcy)

Comment 21 Worker Ant 2016-12-13 02:55:29 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#31) for review on master by Jeff Darcy (jdarcy)

Comment 22 Worker Ant 2016-12-13 04:03:30 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#32) for review on master by Jeff Darcy (jdarcy)

Comment 23 Worker Ant 2016-12-13 13:08:05 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#33) for review on master by Jeff Darcy (jdarcy)

Comment 24 Worker Ant 2016-12-14 17:29:29 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#34) for review on master by Jeff Darcy (jdarcy)

Comment 25 Worker Ant 2016-12-14 17:33:28 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#35) for review on master by Jeff Darcy (jdarcy)

Comment 26 Worker Ant 2016-12-14 17:55:02 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#36) for review on master by Jeff Darcy (jdarcy)

Comment 27 Worker Ant 2016-12-14 18:59:38 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#37) for review on master by Jeff Darcy (jdarcy)

Comment 28 Worker Ant 2016-12-14 21:42:07 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#38) for review on master by Jeff Darcy (jdarcy)

Comment 29 Worker Ant 2016-12-15 03:00:02 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#39) for review on master by Jeff Darcy (jdarcy)

Comment 30 Worker Ant 2016-12-16 21:30:10 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#40) for review on master by Jeff Darcy (jdarcy)

Comment 31 Worker Ant 2016-12-20 20:51:13 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#41) for review on master by Jeff Darcy (jdarcy)

Comment 32 Worker Ant 2016-12-21 18:16:24 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#42) for review on master by Jeff Darcy (jdarcy)

Comment 33 Worker Ant 2017-01-09 16:19:47 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#43) for review on master by Jeff Darcy (jdarcy)

Comment 34 Worker Ant 2017-01-09 16:19:56 UTC
REVIEW: http://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#10) for review on master by Jeff Darcy (jdarcy)

Comment 35 Worker Ant 2017-01-10 22:28:11 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#44) for review on master by Jeff Darcy (jdarcy)

Comment 36 Worker Ant 2017-01-11 18:11:52 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#45) for review on master by Jeff Darcy (jdarcy)

Comment 37 Worker Ant 2017-01-12 18:30:55 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#46) for review on master by Jeff Darcy (jdarcy)

Comment 38 Worker Ant 2017-01-12 19:15:31 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#5) for review on master by Jeff Darcy (jdarcy)

Comment 39 Worker Ant 2017-01-12 22:17:09 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#6) for review on master by Jeff Darcy (jdarcy)

Comment 40 Worker Ant 2017-01-13 00:27:50 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#7) for review on master by Jeff Darcy (jdarcy)

Comment 41 Worker Ant 2017-01-13 04:14:16 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#8) for review on master by Jeff Darcy (jdarcy)

Comment 42 Worker Ant 2017-01-13 04:43:10 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#47) for review on master by Jeff Darcy (jdarcy)

Comment 43 Worker Ant 2017-01-13 18:52:23 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#48) for review on master by Jeff Darcy (jdarcy)

Comment 44 Worker Ant 2017-01-13 18:53:13 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#9) for review on master by Jeff Darcy (jdarcy)

Comment 45 Worker Ant 2017-01-17 16:21:43 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#49) for review on master by Jeff Darcy (jdarcy)

Comment 46 Worker Ant 2017-01-17 16:22:04 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#10) for review on master by Jeff Darcy (jdarcy)

Comment 47 Worker Ant 2017-01-17 22:42:14 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#50) for review on master by Jeff Darcy (jdarcy)

Comment 48 Worker Ant 2017-01-17 22:44:26 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#11) for review on master by Jeff Darcy (jdarcy)

Comment 49 Worker Ant 2017-01-18 14:04:55 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#51) for review on master by Jeff Darcy (jdarcy)

Comment 50 Worker Ant 2017-01-18 14:05:11 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#12) for review on master by Jeff Darcy (jdarcy)

Comment 51 Worker Ant 2017-01-18 17:30:20 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#52) for review on master by Jeff Darcy (jdarcy)

Comment 52 Worker Ant 2017-01-18 19:29:32 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#13) for review on master by Jeff Darcy (jdarcy)

Comment 53 Worker Ant 2017-01-19 21:33:18 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#53) for review on master by Jeff Darcy (jdarcy)

Comment 54 Worker Ant 2017-01-19 21:50:50 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#14) for review on master by Jeff Darcy (jdarcy)

Comment 55 Worker Ant 2017-01-20 01:29:03 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#15) for review on master by Jeff Darcy (jdarcy)

Comment 56 Worker Ant 2017-01-20 17:47:43 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#54) for review on master by Jeff Darcy (jdarcy)

Comment 57 Worker Ant 2017-01-20 20:23:01 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#55) for review on master by Jeff Darcy (jdarcy)

Comment 58 Worker Ant 2017-01-20 22:33:40 UTC
REVIEW: http://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#56) for review on master by Jeff Darcy (jdarcy)

Comment 59 Worker Ant 2017-01-20 22:34:09 UTC
REVIEW: http://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#16) for review on master by Jeff Darcy (jdarcy)

Comment 60 Worker Ant 2017-01-24 12:29:58 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#59) for review on master by Jeff Darcy (jdarcy)

Comment 61 Worker Ant 2017-01-24 17:48:14 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#60) for review on master by Jeff Darcy (jdarcy)

Comment 62 Worker Ant 2017-01-25 03:24:45 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#61) for review on master by Jeff Darcy (jdarcy)

Comment 63 Worker Ant 2017-01-25 04:46:08 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#62) for review on master by Jeff Darcy (jdarcy)

Comment 64 Worker Ant 2017-01-25 13:54:06 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#63) for review on master by Jeff Darcy (jdarcy)

Comment 65 Worker Ant 2017-01-25 14:34:42 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#64) for review on master by Jeff Darcy (jdarcy)

Comment 66 Worker Ant 2017-01-25 15:19:16 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#65) for review on master by Jeff Darcy (jdarcy)

Comment 67 Worker Ant 2017-01-25 18:25:02 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#66) for review on master by Jeff Darcy (jdarcy)

Comment 68 Worker Ant 2017-01-26 02:27:52 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#67) for review on master by Jeff Darcy (jdarcy)

Comment 69 Worker Ant 2017-01-26 16:15:40 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#68) for review on master by Jeff Darcy (jdarcy)

Comment 70 Worker Ant 2017-01-26 17:44:18 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#69) for review on master by Jeff Darcy (jdarcy)

Comment 71 Worker Ant 2017-01-26 18:05:05 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#70) for review on master by Jeff Darcy (jdarcy)

Comment 72 Worker Ant 2017-01-26 20:18:30 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#71) for review on master by Jeff Darcy (jdarcy)

Comment 73 Worker Ant 2017-01-26 20:25:58 UTC
REVIEW: https://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#18) for review on master by Jeff Darcy (jdarcy)

Comment 74 Worker Ant 2017-01-26 21:32:53 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#72) for review on master by Jeff Darcy (jdarcy)

Comment 75 Worker Ant 2017-01-26 21:33:16 UTC
REVIEW: https://review.gluster.org/16363 (multiple: combo patch with multiplexing enabled) posted (#19) for review on master by Jeff Darcy (jdarcy)

Comment 76 Worker Ant 2017-01-27 22:43:36 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#73) for review on master by Jeff Darcy (jdarcy)

Comment 77 Worker Ant 2017-01-28 01:04:07 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#74) for review on master by Jeff Darcy (jdarcy)

Comment 78 Worker Ant 2017-01-28 12:16:56 UTC
REVIEW: https://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#11) for review on master by Jeff Darcy (jdarcy)

Comment 79 Worker Ant 2017-01-30 18:18:08 UTC
REVIEW: https://review.gluster.org/14763 (core: run many bricks within one glusterfsd process) posted (#75) for review on master by Jeff Darcy (jdarcy)

Comment 80 Worker Ant 2017-01-30 19:30:52 UTC
REVIEW: https://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#12) for review on master by Jeff Darcy (jdarcy)

Comment 81 Worker Ant 2017-01-31 00:14:02 UTC
COMMIT: https://review.gluster.org/14763 committed in master by Vijay Bellur (vbellur) 
------
commit 1a95fc3036db51b82b6a80952f0908bc2019d24a
Author: Jeff Darcy <jdarcy>
Date:   Thu Dec 8 16:24:15 2016 -0500

    core: run many bricks within one glusterfsd process
    
    This patch adds support for multiple brick translator stacks running
    in a single brick server process.  This reduces our per-brick memory usage by
    approximately 3x, and our appetite for TCP ports even more.  It also creates
    potential to avoid process/thread thrashing, and to improve QoS by scheduling
    more carefully across the bricks, but realizing that potential will require
    further work.
    
    Multiplexing is controlled by the "cluster.brick-multiplex" global option.  By
    default it's off, and bricks are started in separate processes as before.  If
    multiplexing is enabled, then *compatible* bricks (mostly those with the same
    transport options) will be started in the same process.
    
    Change-Id: I45059454e51d6f4cbb29a4953359c09a408695cb
    BUG: 1385758
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/14763
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 82 Worker Ant 2017-02-01 21:42:20 UTC
REVIEW: https://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#13) for review on master by Jeff Darcy (jdarcy)

Comment 83 Worker Ant 2017-02-02 06:18:49 UTC
REVIEW: https://review.gluster.org/16509 (glusterd: double-check whether brick is alive for stats) posted (#2) for review on master by Jeff Darcy (jdarcy)

Comment 84 Worker Ant 2017-02-02 12:45:17 UTC
REVIEW: https://review.gluster.org/15645 (libglusterfs: make memory pools more thread-friendly) posted (#14) for review on master by Jeff Darcy (jdarcy)

Comment 85 Worker Ant 2017-02-02 12:47:16 UTC
REVIEW: https://review.gluster.org/16510 (socket: retry connect immediately if it fails) posted (#2) for review on master by Jeff Darcy (jdarcy)

Comment 86 Worker Ant 2017-02-02 15:24:41 UTC
REVIEW: https://review.gluster.org/16527 (tests: fix online_brick_count for multiplexing) posted (#1) for review on master by Jeff Darcy (jdarcy)

Comment 87 Worker Ant 2017-02-02 15:49:04 UTC
REVIEW: https://review.gluster.org/16527 (tests: fix online_brick_count for multiplexing) posted (#2) for review on master by Jeff Darcy (jdarcy)

Comment 88 Worker Ant 2017-02-02 17:05:14 UTC
REVIEW: https://review.gluster.org/16528 (tests: use kill_brick instead of kill -9) posted (#2) for review on master by Jeff Darcy (jdarcy)

Comment 89 Worker Ant 2017-02-02 17:35:05 UTC
COMMIT: https://review.gluster.org/16509 committed in master by Atin Mukherjee (amukherj) 
------
commit 72452e07fcf91627afb6d45b921dfefd2610686f
Author: Jeff Darcy <jdarcy>
Date:   Wed Feb 1 21:54:30 2017 -0500

    glusterd: double-check whether brick is alive for stats
    
    With multiplexing, our tests detach bricks from their host processes
    without glusterd being involved.  Thus, when we ask glusterd to fetch
    profile info, it will try to fetch from a brick that's actually not
    present any more.  While it can handle the process being dead and its
    RPC connection being closed, it barfs if it gets a negative response
    from a live brick process.  This is not a problem in normal use,
    because the brick can't disappear without glusterd seeing it.  The fix
    is to double check that the brick is actually running, by looking for
    its pidfile which the tests *do* clean up as part of killing a brick.
    
    Change-Id: I098465b175ecf23538bd7207357c752a2bba8f4e
    BUG: 1385758
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/16509
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Atin Mukherjee <amukherj>

Comment 90 Worker Ant 2017-02-02 18:10:54 UTC
REVIEW: https://review.gluster.org/16529 (glusterd: double-check brick liveness for remove-brick validation) posted (#1) for review on master by Jeff Darcy (jdarcy)

Comment 91 Worker Ant 2017-02-02 18:30:23 UTC
COMMIT: https://review.gluster.org/15645 committed in master by Shyamsundar Ranganathan (srangana) 
------
commit ae47befebeda2de5fd2d706090cbacf4ef60c785
Author: Jeff Darcy <jdarcy>
Date:   Fri Oct 14 10:04:07 2016 -0400

    libglusterfs: make memory pools more thread-friendly
    
    Early multiplexing tests revealed *massive* contention on certain
    pools' global locks - especially for dictionaries and secondarily for
    call stubs.  For the thread counts that multiplexing can create, a
    more lock-free solution is clearly needed.  Also, the current mem-pool
    implementation does a poor job releasing memory back to the system,
    artificially inflating memory usage to match whatever the worst case
    was since the process started.  This is bad in general, but especially
    so for multiplexing where there are more pools and a major point of
    the whole exercise is to reduce memory consumption.
    
    The basic ideas for the new design are these
    
      There is one pool, globally, for each power-of-two size range.
      Every attempt to create a new pool within this range will instead
      add a reference to the existing pool.
    
      Instead of adding pools for each translator within each multiplexed
      brick (potentially infinite and quite possibly thousands), we
      allocate one set of size-based pools per *thread* (hundreds at
      worst).
    
      Each per-thread pool is divided into hot and cold lists.  Every
      allocation first attempts to use the hot list, then the cold list.
      When objects are freed, they always go on the hot list.
    
      There is one global "pool sweeper" thread, which periodically
      reclaims everything in each pool's cold list and then "demotes" the
      current hot list to be the new cold list.
    
      For normal allocation activity, only a per-thread lock need be
      taken, and even that only to guard against very rare contention from
      the pool sweeper.  When threads start and stop, a global lock must
      be taken to add them to the pool sweeper's list.  Lock contention is
      therefore extremely low, and the hot/cold lists also provide good
      locality.
    
    A more complete explanation (of a similar earlier design) can be found
    here:
    
     http://www.gluster.org/pipermail/gluster-devel/2016-October/051160.html
    
    Change-Id: I5bc8a1ba57cfb553998f979a498886e0d006e665
    BUG: 1385758
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/15645
    Reviewed-by: Xavier Hernandez <xhernandez>
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana>

Comment 92 Worker Ant 2017-02-02 19:37:21 UTC
REVIEW: https://review.gluster.org/16527 (tests: fix online_brick_count for multiplexing) posted (#3) for review on master by Jeff Darcy (jdarcy)

Comment 93 Worker Ant 2017-02-02 20:21:40 UTC
COMMIT: https://review.gluster.org/16510 committed in master by Shyamsundar Ranganathan (srangana) 
------
commit 5a57c1592a34ee6632ca1fb38e076dde381d1ae2
Author: Jeff Darcy <jdarcy>
Date:   Wed Feb 1 22:00:32 2017 -0500

    socket: retry connect immediately if it fails
    
    Previously we relied on a complex dance of setting flags, shutting
    down the socket, tearing stuff down, getting an event, tearing more
    stuff down, and waiting for a higher-level retry.  What we really
    need, in the case where we're just trying to connect prematurely e.g.
    to a brick that hasn't fully come up yet, is a simple retry of the
    connect(2) call.
    
    This was discovered by observing failures in ec-new-entry.t with
    multiplexing enabled, but probably fixes other random failures as
    well.
    
    Change-Id: Ibedb8942060bccc96b02272a333c3002c9b77d4c
    BUG: 1385758
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/16510
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana>

Comment 94 Worker Ant 2017-02-02 20:46:23 UTC
COMMIT: https://review.gluster.org/16528 committed in master by Jeff Darcy (jdarcy) 
------
commit 1e849f616f6ae2f97ab307abaff69182c87571ca
Author: Jeff Darcy <jdarcy>
Date:   Thu Feb 2 11:41:06 2017 -0500

    tests: use kill_brick instead of kill -9
    
    The system actually handles this OK, but with multiplexing the result
    of killing the whole process is not what some tests assumed.
    
    Change-Id: I89ebf0039ab1369f25b0bfec3710ec4c13725915
    BUG: 1385758
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/16528
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Vijay Bellur <vbellur>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 95 Worker Ant 2017-02-03 04:06:13 UTC
COMMIT: https://review.gluster.org/16529 committed in master by Atin Mukherjee (amukherj) 
------
commit b4aac5d334d5102ef3b935a200502cfcd63bd12c
Author: Jeff Darcy <jdarcy>
Date:   Thu Feb 2 13:08:04 2017 -0500

    glusterd: double-check brick liveness for remove-brick validation
    
    Same problem as https://review.gluster.org/#/c/16509/ in a different
    place.  Tests detach bricks without glusterd's knowledge, so
    glusterd's internal brick state is out of date and we have to re-check
    (via the brick's pidfile) as well.
    
    BUG: 1385758
    Change-Id: I169538c1c62d72a685a49d57ef65fb6c3db6eab2
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/16529
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Atin Mukherjee <amukherj>

Comment 96 Worker Ant 2017-02-07 18:55:49 UTC
REVIEW: https://review.gluster.org/16544 (tests: keep snapshot bricks separate from regular ones) posted (#2) for review on master by Jeff Darcy (jdarcy)

Comment 97 Worker Ant 2017-02-08 03:22:23 UTC
COMMIT: https://review.gluster.org/16527 committed in master by Vijay Bellur (vbellur) 
------
commit 4a7fd196d4a141f2b693d5b49995733f6ad1776f
Author: Jeff Darcy <jdarcy>
Date:   Thu Feb 2 10:22:00 2017 -0500

    tests: fix online_brick_count for multiplexing
    
    The number of brick processes no longer matches the number of bricks,
    therefore counting processes doesn't work.  Counting *pidfiles* does.
    Ironically, the fix broke multiplex.t which used this function, so it
    now uses a different function with the old process-counting behavior.
    Also had to fix online_brick_count and kill_node in cluster.rc to be
    consistent with the new reality.
    
    Change-Id: I4e81a6633b93227e10604f53e18a0b802c75cbcc
    BUG: 1385758
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/16527
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 98 Worker Ant 2017-02-08 14:44:12 UTC
REVIEW: https://review.gluster.org/16544 (glusterd: keep snapshot bricks separate from regular ones) posted (#3) for review on master by Jeff Darcy (jdarcy)

Comment 99 Worker Ant 2017-02-09 14:43:18 UTC
REVIEW: https://review.gluster.org/16544 (glusterd: keep snapshot bricks separate from regular ones) posted (#4) for review on master by Jeff Darcy (jdarcy)

Comment 100 Worker Ant 2017-02-09 14:44:58 UTC
REVIEW: https://review.gluster.org/16544 (glusterd: keep snapshot bricks separate from regular ones) posted (#5) for review on master by Jeff Darcy (jdarcy)

Comment 101 Worker Ant 2017-02-09 15:56:34 UTC
REVIEW: https://review.gluster.org/16544 (glusterd: keep snapshot bricks separate from regular ones) posted (#6) for review on master by Jeff Darcy (jdarcy)

Comment 102 Worker Ant 2017-02-10 13:16:51 UTC
COMMIT: https://review.gluster.org/16544 committed in master by Jeff Darcy (jdarcy) 
------
commit f1c6ae24361b1bf39794a34ea35a0202a6b49fa6
Author: Jeff Darcy <jdarcy>
Date:   Fri Feb 3 10:51:21 2017 -0500

    glusterd: keep snapshot bricks separate from regular ones
    
    The problem here is that a volume's transport options can change, but
    any snapshots' bricks don't follow along even though they're now
    incompatible (with respect to multiplexing).  This was causing the
    USS+SSL test to fail.  By keeping the snapshot bricks separate
    (though still potentially multiplexed with other snapshot bricks
    including those for other volumes) we can ensure that they remain
    unaffected by changes to their parent volumes.
    
    Also fixed various issues with how the test waits (or more precisely
    didn't) for various events to complete before it continues.
    
    Change-Id: Iab4a8a44fac5760373fac36956a3bcc27cf969da
    BUG: 1385758
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: https://review.gluster.org/16544
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Avra Sengupta <asengupt>
    Tested-by: Avra Sengupta <asengupt>

Comment 103 Jeff Darcy 2017-03-08 15:28:39 UTC
*** Bug 1360401 has been marked as a duplicate of this bug. ***

Comment 104 Jeff Darcy 2017-03-08 15:30:31 UTC
Any future work on this can be done as separate bugs.

Comment 105 Shyamsundar 2017-05-30 18:35:29 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report.

glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.