Bug 469920

> 3. Ensure that the qpid process can lock enough physical memory. With current
> default connection settings each connection will use 2MB of locked memory in
> the broker. This can be achieved with the ulimit command, and the
> /etc/security/limits.comf file.

I got that wrong - the current default settings for the client lib will actually allocate 128 buffers of 64k each using a total of 8MB per connection on both the broker and client.

Currently the 128 buffers (64 read/64 write) is fixed but 64k buffer size is dependent on the specified maximum buffer size in the connection options so could be made smaller.

This information is still incomplete. It's useful, but we need to describe the feature and capture a process before introducing a lot of these concepts. Simply put, I would prefer to have something in the following structure:

RDMA/Infiniband

* Quick overview of RDMA/Infiniband - what problems does it solve, how (briefly, high-level) does it work?

* Methods of using RDMA/Infiniband - what are the important things to know, what are the main methods of operation

* PROCESS - how to get it up and running, including config options and commands

* Look out for ...

I'm leaving it out of the documentation for now, once we have some more solid info we can get it added in pretty quickly.

Thanks, and apologies for the late notice on this one.

LKB

Sorry I've no time to do this, and I probably not the best person to do the background stuff anyway. I'm the expect on the implementation details not really on why you'd use rdma/infiniband in the first place.

In the meantime I'm raising a BZ for some release notes that talk about setting rdma up if you need it.

Are you able to provide this info for 1.2?

LKB

*** Bug 480216 has been marked as a duplicate of this bug. ***

Here's my first shot:

MRG  Messaging connections  can  use Infiniband,  which provides  high
speed point-to-point  bidirectional serial links that  are much faster
than TCP/IP connections.

The machines running the server and client must each have Infiniband
properly installed. In particular:

- The kernel driver and the user space driver for your Infiniband
  hardware must both be installed.

  See http://kbase.redhat.com/faq/docs/DOC-22217 for a list of
  hardware supported on RHEL 4 and RHEL 5, and the kernel and user
  space drivers associated with each.

- Allocate lockable memory for Infiniband. 

  By default, the operating system can swap out all user
  memory. Infiniband requires lockable memory, which can not be
  swapped out. Each connection requires 8 Megabytes (8192 bytes) of
  lockable memory.

  To allocate lockable memory, edit /etc/security/limits.conf to set
  the soft limit, which is the initial amount of lockable memory that
  is allocated, and the hard limit, which is the maximum amount that
  will be allocated.

- The Infiniband interface must be configured to allow IP over
  Infiniband. This is not used for message transport, but is
  nevertheless needed for RDMA connection management.

* To enable Infiniband on the Qpid server:

- Make sure that the RDMA plugin, rdma.so, is present in the plugins
  directory.

- Allocate lockable memory for Infiniband. 

  If the user running the server is qpidd, the soft limit should
  support 64 connections (64*8192=524288), and the hard limit should
  support 256 connections (256*8192=2097152), add these entries:

  qpidd soft memlock 524288
  qpdd hard memlock 2097152


* To use Infiniband in a Qpid messaging client:

- Make sure that the package qpidc-rdma has been installed. If it has,
  the file dma-connector.so will be present in the plugins directory.

- Allocate lockable memory for Infiniband.   

  If the limit is set for all users, the soft limit should support 16
  connections (16*8192=32768), and the hard limit should support 64
  connections (64*8192=524288), add these entries:

  * soft memlock 32768
  * hard memlock 524288

  If you want to set a limit for a particular user, use the UID for
  that user when setting the limits:

  andrew soft memlock 32768
  andrew hard memlock 524288

In para 1:

"that  are much faster than TCP/IP connections"

this isn't necessarily true, how about "that can be faster and have much lower latency than TCP connections"

In para 6:

"the soft limit, which is the initial amount of lockable memory that is allocated"

not really true: it's the initial limit that is assigned, ie the soft limit is the max amount of memory you can lock when you log in unless you change the limit; the hard limit is the maximum you're allowed to set the limit to. How about " which is the initial amount of lockable memory that can be allocated per process"

Note that the limits are per process and not per system. So I'm not sure your recommendation of 16 connections for clients is necessary. However the limit is mainly there to protect the system in case of program error and the memory isn't locked unless it's needed, so there's not much danger in allowing more than you need.

Once you have sized the lockable memory yourself, there is no point in setting a different hard and soft limit. Both are present mainly for interactive use.

[An enhancement request might be to have the rdma module check that amount of lockable memory allowed and try to up the amount if the hard limit allowed and it needed, but it doesn't do anything like that now]