We need to change the epoll thread model and make it lighter. http://review.gluster.com/3842 is one such approach.
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#1) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#2) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#3) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#4) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#5) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#6) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#7) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#8) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#10) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#9) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/8098 (epoll: Handle client and server FDs in a separate event pool) posted (#10) for review on master by Vijaikumar Mallikarjuna (vmallika)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#11) for review on master by Harshavardhana (harsha)
REVIEW: http://review.gluster.org/3842 (test: multi-threaded edge-triggered epoll plus *one other fix*) posted (#12) for review on master by Jeff Darcy (jdarcy)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#13) for review on master by Jeff Darcy (jdarcy)
REVIEW: http://review.gluster.org/8971 (epoll: edge triggered and multi-threaded epoll) posted (#1) for review on master by Jeff Darcy (jdarcy)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#14) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#15) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#16) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#17) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#3) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#18) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#19) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#4) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#20) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#5) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#21) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#6) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/3842 (epoll: edge triggered and multi-threaded epoll) posted (#22) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/9488 (epoll: Adding the ability to configure epoll threads) posted (#7) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/9591 (protocol/client: sequence CHILD_UP, CHILD_DOWN etc notifications) posted (#4) for review on master by Krishnan Parthasarathi (kparthas)
COMMIT: http://review.gluster.org/3842 committed in master by Vijay Bellur (vbellur) ------ commit c61074400a45e69c6edbf82b8ed02568726d37ae Author: Vijaikumar M <vmallika> Date: Thu Jun 19 15:41:22 2014 +0530 epoll: edge triggered and multi-threaded epoll - edge triggered (oneshot) polling with epoll - pick one event to avoid multiple events getting picked up by same thread and so get better distribution of events against multiple threads - wire support for multiple poll threads to epoll_wait in parallel - evdata to store absolute index and not hint for epoll - store index and gen of slot instead of fd and index hint - perform fd close asynchronously inside event.c for multithread safety - poll is still single threaded Change-Id: I536851dda0ab224c5d5a1b130a571397c9cace8f BUG: 1104462 Signed-off-by: Anand Avati <avati> Signed-off-by: Vijaikumar M <vmallika> Signed-off-by: Jeff Darcy <jdarcy> Signed-off-by: Shyam <srangana> Reviewed-on: http://review.gluster.org/3842 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/9488 committed in master by Vijay Bellur (vbellur) ------ commit a7f5893c9243c8c563db215352fa7e47f6968e8b Author: Shyam <srangana> Date: Mon Jan 26 14:20:31 2015 -0500 epoll: Adding the ability to configure epoll threads Add the ability to configure the number of event threads for various gluster services. Currently with the multi thread epoll patch, it is possible to have more than one thread waiting on socket activity and processing the same. This thread count is currently static, which this commit makes dynamic. The current services which use IO path, i.e brick processes, any client process (nfs, FUSE, gfapi, heal, rebalance, etc.a), gain 2 set parameters to control the number of threads that are processing events. These settings are, - client.event-threads <n> - server.event-threads <n> The client setting affects the client graph consumers, and the server setting affects the brick processes. These are processed and inited/reconfigured using the client/server protocol xlators. Other services (say glusterd) would need to extend similar configuration settings to take advantage of multi threaded event processing. At present glusterd is not enabled with this commit, as it does not stand to gain from this multi-threading (as I understand it). Change-Id: Id8422fc57a9f95a135158eb6477ccf9d3c9ea4d9 BUG: 1104462 Signed-off-by: Shyam <srangana> Reviewed-on: http://review.gluster.org/9488 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/9591 committed in master by Vijay Bellur (vbellur) ------ commit f18a3f30bbeaf3bb067b913082830d7f874555ca Author: Krishnan Parthasarathi <kparthas> Date: Thu Feb 5 15:41:35 2015 +0530 protocol/client: sequence CHILD_UP, CHILD_DOWN etc notifications ... from all bricks in the volume This patch is important in the context of MT epoll. With MT epoll, notification events from client xlators could reach cluster xlators like afr, dht, ec, stripe etc. in different orders. For e.g, In a distributed replicate volume of 2 bricks, namely Brick1 and Brick2, the following network events are observed by a mount process. - connection to Brick1 is broken. - connection to Brick1 has been restored. - connection to Brick2 is broken. - connection to Brick2 has been restored. Without establishing a total ordering of events, we can't guarantee that cluster xlators like afr, dht perceive them in the same order. While we would expect afr (say) to perceive it as only one of Brick1 and Brick2 going down at any given time, it is possible for the notification of Brick2 going offline to race with the notification of Brick1 coming back online. Change-Id: I78f5a52bfb05593335d0e9ad53ebfff98995593d BUG: 1104462 Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/9591 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/9604 (epoll: Initialize server/client xlator event-threads correctly) posted (#1) for review on master by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/9604 (epoll: Initialize server/client xlator event-threads correctly) posted (#2) for review on master by Shyamsundar Ranganathan (srangana)
COMMIT: http://review.gluster.org/9604 committed in master by Vijay Bellur (vbellur) ------ commit fe73f12f6ef2deb402b622747cad5e82b11e57af Author: Shyam <srangana> Date: Sun Feb 8 20:08:46 2015 -0500 epoll: Initialize server/client xlator event-threads correctly In the patch http://review.gluster.org/#/c/9488/ the default configuration for server and client protocol xlators, had event-threads set to 2, but the default conf was not updated with these values, hence the default threads remained at 1. This patch corrects the same by first updating the thread count from the default configuration before updating it with the latest. Change-Id: I3ce54053a59ca796b004fa5462e43ca19a5f2915 BUG: 1104462 Signed-off-by: Shyam <srangana> Reviewed-on: http://review.gluster.org/9604 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> Reviewed-by: Krishnan Parthasarathi <kparthas> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/9648 (client/server: fix reconfigure for event-threads option) posted (#1) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/9648 (client/server: fix reconfigure for event-threads option) posted (#2) for review on master by Krishnan Parthasarathi (kparthas)
COMMIT: http://review.gluster.org/9648 committed in master by Raghavendra G (rgowdapp) ------ commit ee9b0aab614b56aa8f3c8ed56ced88f769c0ee47 Author: Krishnan Parthasarathi <kparthas> Date: Thu Feb 12 17:29:28 2015 +0530 client/server: fix reconfigure for event-threads option Change-Id: Ica8bca13e4feb941e22651b642b848be165ccc9e BUG: 1104462 Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/9648 Reviewed-by: Shyamsundar Ranganathan <srangana> Reviewed-by: Pranith Kumar Karampuri <pkarampu> Reviewed-by: Atin Mukherjee <amukherj> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user