Bug 1104462 - RFC: make epoll multithreaded
Summary: RFC: make epoll multithreaded
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: glusterd
Version: mainline
Hardware: x86_64
OS: Unspecified
low
low
Target Milestone: ---
Assignee: Vijaikumar Mallikarjuna
QA Contact:
URL:
Whiteboard:
Depends On: 1096729 1104459
Blocks: 1088355 1097224 1098021 1098025 1194166
TreeView+ depends on / blocked
 
Reported: 2014-06-04 04:58 UTC by ssamanta
Modified: 2016-05-11 22:47 UTC (History)
11 users (show)

Fixed In Version: glusterfs-3.7.0
Doc Type: Bug Fix
Doc Text:
Clone Of: 1104459
Environment:
Last Closed: 2015-05-14 17:25:50 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Comment 1 Vijaikumar Mallikarjuna 2014-06-10 07:37:54 UTC
We need to change the epoll thread model and make it lighter.

http://review.gluster.com/3842 is one such approach.

Comment 2 Anand Avati 2014-06-18 08:53:17 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#1) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 3 Anand Avati 2014-06-18 09:23:07 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#2) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 4 Anand Avati 2014-06-18 09:28:26 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#3) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 5 Anand Avati 2014-06-18 09:51:16 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#4) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 6 Anand Avati 2014-06-18 09:53:47 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#5) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 7 Anand Avati 2014-06-18 10:15:31 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#6) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 8 Anand Avati 2014-06-19 08:56:00 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#7) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 9 Anand Avati 2014-06-19 09:18:51 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#8) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 10 Anand Avati 2014-06-19 10:43:16 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#10) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 11 Anand Avati 2014-06-20 07:08:44 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#9) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 12 Anand Avati 2014-06-27 06:23:58 UTC
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#10) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 13 Anand Avati 2014-09-04 22:19:01 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#11) for review on master by Harshavardhana (harsha)

Comment 14 Anand Avati 2014-10-22 19:02:07 UTC
REVIEW: http://review.gluster.org/3842 (test: multi-threaded edge-triggered epoll plus *one other fix*) posted (#12) for review on master by Jeff Darcy (jdarcy)

Comment 15 Anand Avati 2014-10-22 19:09:53 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#13) for review on master by Jeff Darcy (jdarcy)

Comment 16 Anand Avati 2014-10-23 13:42:43 UTC
REVIEW: http://review.gluster.org/8971 (epoll: edge triggered and multi-threaded epoll) posted (#1) for review on master by Jeff Darcy (jdarcy)

Comment 17 Anand Avati 2015-01-21 20:01:27 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#14) for review on master by Shyamsundar Ranganathan (srangana)

Comment 18 Anand Avati 2015-01-26 19:11:33 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#15) for review on master by Shyamsundar Ranganathan (srangana)

Comment 19 Anand Avati 2015-01-26 19:31:01 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#16) for review on master by Shyamsundar Ranganathan (srangana)

Comment 20 Anand Avati 2015-01-28 20:52:05 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#17) for review on master by Shyamsundar Ranganathan (srangana)

Comment 21 Anand Avati 2015-01-29 20:19:44 UTC
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#3) for review on master by Shyamsundar Ranganathan (srangana)

Comment 22 Anand Avati 2015-01-29 20:19:48 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#18) for review on master by Shyamsundar Ranganathan (srangana)

Comment 23 Anand Avati 2015-01-30 19:31:53 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#19) for review on master by Shyamsundar Ranganathan (srangana)

Comment 24 Anand Avati 2015-01-30 19:40:11 UTC
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#4) for review on master by Shyamsundar Ranganathan (srangana)

Comment 25 Anand Avati 2015-02-03 20:03:56 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#20) for review on master by Shyamsundar Ranganathan (srangana)

Comment 26 Anand Avati 2015-02-03 20:05:29 UTC
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#5) for review on master by Shyamsundar Ranganathan (srangana)

Comment 27 Anand Avati 2015-02-04 21:09:55 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#21) for review on master by Shyamsundar Ranganathan (srangana)

Comment 28 Anand Avati 2015-02-04 21:10:07 UTC
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#6) for review on master by Shyamsundar Ranganathan (srangana)

Comment 29 Anand Avati 2015-02-05 16:06:50 UTC
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#22) for review on master by Shyamsundar Ranganathan (srangana)

Comment 30 Anand Avati 2015-02-05 16:42:25 UTC
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#7) for review on master by Shyamsundar Ranganathan (srangana)

Comment 31 Anand Avati 2015-02-06 07:43:19 UTC
REVIEW: http://review.gluster.org/9591 (protocol/client: sequence CHILD_UP, CHILD_DOWN etc notifications) posted (#4) for review on master by Krishnan Parthasarathi (kparthas)

Comment 32 Anand Avati 2015-02-07 21:17:34 UTC
COMMIT: http://review.gluster.org/3842 committed in master by Vijay Bellur (vbellur) 
------
commit c61074400a45e69c6edbf82b8ed02568726d37ae
Author: Vijaikumar M <vmallika>
Date:   Thu Jun 19 15:41:22 2014 +0530

    epoll: edge triggered and multi-threaded epoll
    
    - edge triggered (oneshot) polling with epoll
    - pick one event to avoid multiple events getting picked up by same
      thread
      and so get better distribution of events against multiple threads
    - wire support for multiple poll threads to epoll_wait in parallel
    - evdata to store absolute index and not hint for epoll
    - store index and gen of slot instead of fd and index hint
    - perform fd close asynchronously inside event.c for multithread safety
    - poll is still single threaded
    
    Change-Id: I536851dda0ab224c5d5a1b130a571397c9cace8f
    BUG: 1104462
    Signed-off-by: Anand Avati <avati>
    Signed-off-by: Vijaikumar M <vmallika>
    Signed-off-by: Jeff Darcy <jdarcy>
    Signed-off-by: Shyam <srangana>
    Reviewed-on: http://review.gluster.org/3842
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 33 Anand Avati 2015-02-07 21:23:07 UTC
COMMIT: http://review.gluster.org/9488 committed in master by Vijay Bellur (vbellur) 
------
commit a7f5893c9243c8c563db215352fa7e47f6968e8b
Author: Shyam <srangana>
Date:   Mon Jan 26 14:20:31 2015 -0500

    epoll: Adding the ability to configure epoll threads
    
    Add the ability to configure the number of event threads
    for various gluster services.
    
    Currently with the multi thread epoll patch, it is possible
    to have more than one thread waiting on socket activity and
    processing the same. This thread count is currently static,
    which this commit makes dynamic.
    
    The current services which use IO path, i.e brick processes,
    any client process (nfs, FUSE, gfapi, heal,
    rebalance, etc.a), gain 2 set parameters to control the number
    of threads that are processing events. These settings are,
      - client.event-threads <n>
      - server.event-threads <n>
    
    The client setting affects the client graph consumers, and the
    server setting affects the brick processes. These are processed
    and inited/reconfigured using the client/server protocol xlators.
    
    Other services (say glusterd) would need to extend similar
    configuration settings to take advantage of multi threaded event
    processing.
    
    At present glusterd is not enabled with this commit, as it does not
    stand to gain from this multi-threading (as I understand it).
    
    Change-Id: Id8422fc57a9f95a135158eb6477ccf9d3c9ea4d9
    BUG: 1104462
    Signed-off-by: Shyam <srangana>
    Reviewed-on: http://review.gluster.org/9488
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 34 Anand Avati 2015-02-07 21:25:29 UTC
COMMIT: http://review.gluster.org/9591 committed in master by Vijay Bellur (vbellur) 
------
commit f18a3f30bbeaf3bb067b913082830d7f874555ca
Author: Krishnan Parthasarathi <kparthas>
Date:   Thu Feb 5 15:41:35 2015 +0530

    protocol/client: sequence CHILD_UP, CHILD_DOWN etc notifications
    
    ... from all bricks in the volume
    
    This patch is important in the context of MT epoll. With MT epoll,
    notification events from client xlators could reach cluster xlators like
    afr, dht, ec, stripe etc. in different orders.
    
    For e.g, In a distributed replicate volume of 2 bricks, namely Brick1
    and Brick2, the following network events are observed by a mount
    process.
    
    - connection to Brick1 is broken.
    - connection to Brick1 has been restored.
    
    - connection to Brick2 is broken.
    - connection to Brick2 has been restored.
    
    Without establishing a total ordering of events, we can't guarantee that
    cluster xlators like afr, dht perceive them in the same order.  While we
    would expect afr (say) to perceive it as only one of Brick1 and Brick2
    going down at any given time, it is possible for the notification of
    Brick2 going offline to race with the notification of Brick1 coming back
    online.
    
    Change-Id: I78f5a52bfb05593335d0e9ad53ebfff98995593d
    BUG: 1104462
    Signed-off-by: Krishnan Parthasarathi <kparthas>
    Reviewed-on: http://review.gluster.org/9591
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 35 Anand Avati 2015-02-09 01:11:30 UTC
REVIEW: http://review.gluster.org/9604 (epoll: Initialize server/client xlator event-threads correctly) posted (#1) for review on master by Shyamsundar Ranganathan (srangana)

Comment 36 Anand Avati 2015-02-09 01:13:15 UTC
REVIEW: http://review.gluster.org/9604 (epoll: Initialize server/client xlator event-threads correctly) posted (#2) for review on master by Shyamsundar Ranganathan (srangana)

Comment 37 Anand Avati 2015-02-09 15:17:31 UTC
COMMIT: http://review.gluster.org/9604 committed in master by Vijay Bellur (vbellur) 
------
commit fe73f12f6ef2deb402b622747cad5e82b11e57af
Author: Shyam <srangana>
Date:   Sun Feb 8 20:08:46 2015 -0500

    epoll: Initialize server/client xlator event-threads correctly
    
    In the patch http://review.gluster.org/#/c/9488/ the default
    configuration for server and client protocol xlators, had
    event-threads set to 2, but the default conf was not updated
    with these values, hence the default threads remained at 1.
    
    This patch corrects the same by first updating the thread count
    from the default configuration before updating it with the
    latest.
    
    Change-Id: I3ce54053a59ca796b004fa5462e43ca19a5f2915
    BUG: 1104462
    Signed-off-by: Shyam <srangana>
    Reviewed-on: http://review.gluster.org/9604
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Atin Mukherjee <amukherj>
    Reviewed-by: Krishnan Parthasarathi <kparthas>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 38 Anand Avati 2015-02-12 12:02:31 UTC
REVIEW: http://review.gluster.org/9648 (client/server: fix reconfigure for event-threads option) posted (#1) for review on master by Krishnan Parthasarathi (kparthas)

Comment 39 Anand Avati 2015-02-12 12:48:17 UTC
REVIEW: http://review.gluster.org/9648 (client/server: fix reconfigure for event-threads option) posted (#2) for review on master by Krishnan Parthasarathi (kparthas)

Comment 40 Anand Avati 2015-02-19 10:12:53 UTC
COMMIT: http://review.gluster.org/9648 committed in master by Raghavendra G (rgowdapp) 
------
commit ee9b0aab614b56aa8f3c8ed56ced88f769c0ee47
Author: Krishnan Parthasarathi <kparthas>
Date:   Thu Feb 12 17:29:28 2015 +0530

    client/server: fix reconfigure for event-threads option
    
    Change-Id: Ica8bca13e4feb941e22651b642b848be165ccc9e
    BUG: 1104462
    Signed-off-by: Krishnan Parthasarathi <kparthas>
    Reviewed-on: http://review.gluster.org/9648
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Reviewed-by: Atin Mukherjee <amukherj>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
    Tested-by: Raghavendra G <rgowdapp>

Comment 41 Niels de Vos 2015-05-14 17:25:50 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 42 Niels de Vos 2015-05-14 17:35:27 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 43 Niels de Vos 2015-05-14 17:37:48 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 44 Niels de Vos 2015-05-14 17:42:18 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.