Rdma pre-allocates a set of buffers and registers them with infiniband to be used to store the incoming data on the wire. Whenever there are no such free buffers on receiver's end, send from the sender will fail. Currently rdma manages this with client and server exchanging the number of recv-buffers they have during handshake. After handshake, with client and server knowing how many buffers the other end have and with the fact the each send would consume a buffer on receiver end (which, the receiver puts back to infiniband once it is done with processing the contents), can implement a quota based mechanism so that we would not run into send failures. However, as of now the queue of receive buffers is stored per-device but not per-transport. Hence the no of buffers advertised by the server to each of the client should be (nuber-of-buffers/number-of-clients - assuming equal allocation of buffers, which need not be the case). Instead we are advertising to each client that the server has all the receive buffers exclusively for the client, which is not the case.

However the problem with sharing a global queue of receive buffers with all the clients is that we don't know upfront how many clients will connect to a server. Hence it'll be difficult to find out how many buffers each client should be allocated. An alternative solution would be to maintain a separate queue of receive buffers for each transport.

regards,
Raghavendra.