Hide Forgot
ib-verbs as of now uses fixed number of send/recv buffers for transporting data using send/recv. This requires copying of data from/to send/recv buffers to/from buffers of translators invoking transport_submit. With introduction of 0-copy rdma we can have a hybrid approach where short messages are sent using send/recv and large buffers (in fops like read/write etc) can be sent using rdma. This also helps in scalability since it reduces dependence on a fixed number of buffers. For rdma, buffers are dynamically allocated. With existing implementation for a setup having thousands of clients, server side should have large number of buffers. As a prerequisite to this, we need variable sized iobufs.
There are some disruptive changes to bring in this change in release-3.0. Its better to bring all the disruptive changes in next 3.1 release. Raghu, finish this feature asap so it can go with alpha release of NFS xlator next week.. Let us know what you think.
PATCH: http://patches.gluster.com/patch/4250 in master (rpcsvc: allow creation of both rdma and socket for a single server transport.)
PATCH: http://patches.gluster.com/patch/4311 in master (rpcsvc: handle the return value of rpcsvc_create_listeners properly.)
PATCH: http://patches.gluster.com/patch/4322 in master (rpcsvc: fix double free happening when listener creation fails.)
PATCH: http://patches.gluster.com/patch/4395 in master (Bring in new transport rdma.)
PATCH: http://patches.gluster.com/patch/4396 in master (changes to client to make use of the rdma functionality.)
PATCH: http://patches.gluster.com/patch/4397 in master (Add rdma as valid transport type for client and server protocol.)
PATCH: http://patches.gluster.com/patch/4408 in master (rpc-transport/rdma: free ioq entry even when churning of message fails.)
PATCH: http://patches.gluster.com/patch/4409 in master (rpc-transport/rdma: By default, enable building of rdma only if libibverbs is installed.)
PATCH: http://patches.gluster.com/patch/4525 in master (rpc-transport/rdma: notify rpc when pollerr happens, so that client side can try to reconnect.)
PATCH: http://patches.gluster.com/patch/4646 in master (rpc-transport/rdma: change the default listening port to 6968.)
PATCH: http://patches.gluster.com/patch/4713 in master (rpc/rpcsvc: fix invalid free in rpcsvc_create_listeners.)
PATCH: http://patches.gluster.com/patch/5192 in master (rpc-transport/rdma: send disconnect event only if the transport is already connected.)
PATCH: http://patches.gluster.com/patch/5467 in master (rpc-transport/rdma: by default use infiniband port 1)
PATCH: http://patches.gluster.com/patch/5480 in master (rpc-transport/rdma: associate mem-pools with device instead of transport.)
PATCH: http://patches.gluster.com/patch/5637 in master (rpc-transport/rdma: Fix 2KB as threshold size for msgs that can be transferred inline.)
We should mention that RDMA complete support is available in 3.1.2 (upto 3.1.1 we had some issues). If added, can mark it DA