Bug 764949 (GLUSTER-3217)

Summary: [FEAT] Design the better Receive-buffer management in RDMA transport
Product: [Community] GlusterFS Reporter: Raghavendra G <raghavendra>
Component: rdmaAssignee: Raghavendra G <rgowdapp>
Status: CLOSED EOL QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: mainlineCC: bugs, gluster-bugs, rwheeler
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-22 11:46:38 EDT Type: ---
Regression: RTNR Mount Type: All
Documentation: DNR CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Raghavendra G 2011-07-22 01:00:09 EDT
Rdma pre-allocates a set of buffers and registers them with infiniband to be used to store the incoming data on the wire. Whenever there are no such free buffers on receiver's end, send from the sender will fail. Currently rdma manages this with client and server exchanging the number of recv-buffers they have during handshake. After handshake, with client and server knowing how many buffers the other end have and with the fact the each send would consume a buffer on receiver end (which, the receiver puts back to infiniband once it is done with processing the contents), can implement a quota based mechanism so that we would not run into send failures. However, as of now the queue of receive buffers is stored per-device but not per-transport. Hence the no of buffers advertised by the server to each of the client should be (nuber-of-buffers/number-of-clients - assuming equal allocation of buffers, which need not be the case). Instead we are advertising to each client that the server has all the receive buffers exclusively for the client, which is not the case.

However the problem with sharing a global queue of receive buffers with all the clients is that we don't know upfront how many clients will connect to a server. Hence it'll be difficult to find out how many buffers each client should be allocated. An alternative solution would be to maintain a separate queue of receive buffers for each transport.

Comment 1 Amar Tumballi 2012-02-27 05:35:55 EST
This is the priority for immediate future (before 3.3.0 GA release). Will bump the priority up once we take RDMA related tasks.
Comment 2 Kaleb KEITHLEY 2015-10-22 11:46:38 EDT
because of the large number of bugs filed against mainline version\ is ambiguous and about to be removed as a choice.

If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.