Bug 1550941

+++ This bug was initially created as a clone of Bug #1549497 +++

Description of problem:
Bottleneck has been introduced due to upstream patch
https://review.gluster.org/17105

To be more responsive to client pings in brick-mux mode, the IO/Management fops to the brick were separated from the ping responder code by designating a single rpcsvc request handler thread per gluster program. This created a bottleneck with multiple event threads queuing request to a queue that was being read by a single request handler thread that dispatched request to the IO thread pool. To alleviate this bottleneck it was necessary to scale rpcsvc request handler threads.

As part of brick-mux implementation the event handler threads were scaled so that there was one event thread serving requests per brick. I'd like to propose patch https://review.gluster.org/19337 for RHGS 3.4.0 to alleviate the bottleneck of a single gluster program thread interfacing with the IO thread pool.

Patch https://review.gluster.org/19337 attempts to continue work on brick-mux to facilitate request processing per gluster program by scaling the gluster program threads to match the event handler threads. The gluster program threads help the event handler threads to delegate the dispatch of the request to the IO handler threads much quicker so that the event handler threads can go back to reading RPC requests quicker than being busy with request hand-off at the brick.