Bug 469920 - Documentation doesn't have any information about using qpid with RDMA protocols
Documentation doesn't have any information about using qpid with RDMA protocols
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: Messaging_Programming_Reference (Show other bugs)
1.1
All Linux
high Severity medium
: 1.3
: ---
Assigned To: Jonathan Robie
Jeff Needle
: Documentation
: 480216 (view as bug list)
Depends On:
Blocks: 480216
  Show dependency treegraph
 
Reported: 2008-11-04 14:00 EST by Andrew Stitcher
Modified: 2013-08-05 20:53 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 480216 (view as bug list)
Environment:
Last Closed: 2011-01-25 16:13:36 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Andrew Stitcher 2008-11-04 14:00:57 EST
The most important issues when setting up qpid to run over an RDMA transport (currently we've only tested over Infiniband):

For daemon:
1. Ensure that qpidd-rdma package is installed. You can check that the qpidd rdma plugin is available by checking for the file /usr/lib[64]/qpid/daemon/plugin/rdma.so.

2. Make sure that IP over IB is configured on the infiniband interface and the interface is assigned an IP address. This is necessary for the RDMA Connection Manager layer to work even though the qpid RDMA transport does not use IP over IB.

3. Ensure that the qpid process can lock enough physical memory. With current default connection settings each connection will use 2MB of locked memory in the broker. This can be achieved with the ulimit command, and the /etc/security/limits.comf file.

- Before release we may introduce an /etc/security/limits.d/qpidd.conf file which will change this limit by default for the qpidd user and hence for the system qpidd.

- If we do we will probably use a limit of 128M (131072) which would be good for 64 connections with the current defaults and more than that would require changing /etc/security/limits.d/qpidd.conf.

For daemon/client:
4. In order for Infiniband to work on any machine the appropriate USER space infiniband driver must be installed. This won't happen automatically. For example to drive the Mellanox IB adaptors the libmthca pkg must be installed even though the kernel space driver will show up as being loaded. This is because a large part of the point of IB is that you use it without going through the kernel at all, and so don't have to context switch. You do however need user space drivers that know how to talk to the card.

For client:
1. For client programs you need the package qpidc-rdma to use the RDMA transport. You can check this is present by seeing whether the file /usr/lib[64]/qpid/client/rdma-connector.so is present.

2. The client programs that are supplied in the qpid-perftest package (or whatever we rename it to) now take an extra parameter to specify the transport mechanism for amqp. this is "-P" or "--protocol". To select the rdma transport use "-P rdma". (this can also be used to select ssl, and defaults to tcp).

3. The client also needs to be able to lock physical memory and this is achieved by editting the /etc/security/limits.conf file on the client system. You can add lines like:
  * soft memlock 32768
  * hard memlock 131072
which would set the softlimit to 32MB and the hardlimit to 128MB for everyone.
Comment 1 Andrew Stitcher 2008-11-05 16:16:11 EST
> 3. Ensure that the qpid process can lock enough physical memory. With current
> default connection settings each connection will use 2MB of locked memory in
> the broker. This can be achieved with the ulimit command, and the
> /etc/security/limits.comf file.

I got that wrong - the current default settings for the client lib will actually allocate 128 buffers of 64k each using a total of 8MB per connection on both the broker and client.

Currently the 128 buffers (64 read/64 write) is fixed but 64k buffer size is dependent on the specified maximum buffer size in the connection options so could be made smaller.
Comment 2 Lana Brindley 2008-11-13 02:47:10 EST
This information is still incomplete. It's useful, but we need to describe the feature and capture a process before introducing a lot of these concepts. Simply put, I would prefer to have something in the following structure:

RDMA/Infiniband

* Quick overview of RDMA/Infiniband - what problems does it solve, how (briefly, high-level) does it work?

* Methods of using RDMA/Infiniband - what are the important things to know, what are the main methods of operation

* PROCESS - how to get it up and running, including config options and commands

* Look out for ...

I'm leaving it out of the documentation for now, once we have some more solid info we can get it added in pretty quickly.

Thanks, and apologies for the late notice on this one.

LKB
Comment 3 Andrew Stitcher 2009-01-15 13:28:57 EST
Sorry I've no time to do this, and I probably not the best person to do the background stuff anyway. I'm the expect on the implementation details not really on why you'd use rdma/infiniband in the first place.

In the meantime I'm raising a BZ for some release notes that talk about setting rdma up if you need it.
Comment 4 Lana Brindley 2009-03-08 21:27:36 EDT
Are you able to provide this info for 1.2?

LKB
Comment 5 Lana Brindley 2009-03-08 21:29:09 EDT
*** Bug 480216 has been marked as a duplicate of this bug. ***
Comment 7 Jonathan Robie 2010-01-28 17:54:08 EST
Here's my first shot:

MRG  Messaging connections  can  use Infiniband,  which provides  high
speed point-to-point  bidirectional serial links that  are much faster
than TCP/IP connections.

The machines running the server and client must each have Infiniband
properly installed. In particular:

- The kernel driver and the user space driver for your Infiniband
  hardware must both be installed.

  See http://kbase.redhat.com/faq/docs/DOC-22217 for a list of
  hardware supported on RHEL 4 and RHEL 5, and the kernel and user
  space drivers associated with each.

- Allocate lockable memory for Infiniband. 

  By default, the operating system can swap out all user
  memory. Infiniband requires lockable memory, which can not be
  swapped out. Each connection requires 8 Megabytes (8192 bytes) of
  lockable memory.

  To allocate lockable memory, edit /etc/security/limits.conf to set
  the soft limit, which is the initial amount of lockable memory that
  is allocated, and the hard limit, which is the maximum amount that
  will be allocated.

- The Infiniband interface must be configured to allow IP over
  Infiniband. This is not used for message transport, but is
  nevertheless needed for RDMA connection management.

* To enable Infiniband on the Qpid server:

- Make sure that the RDMA plugin, rdma.so, is present in the plugins
  directory.

- Allocate lockable memory for Infiniband. 

  If the user running the server is qpidd, the soft limit should
  support 64 connections (64*8192=524288), and the hard limit should
  support 256 connections (256*8192=2097152), add these entries:

  qpidd soft memlock 524288
  qpdd hard memlock 2097152


* To use Infiniband in a Qpid messaging client:

- Make sure that the package qpidc-rdma has been installed. If it has,
  the file dma-connector.so will be present in the plugins directory.

- Allocate lockable memory for Infiniband.   

  If the limit is set for all users, the soft limit should support 16
  connections (16*8192=32768), and the hard limit should support 64
  connections (64*8192=524288), add these entries:

  * soft memlock 32768
  * hard memlock 524288

  If you want to set a limit for a particular user, use the UID for
  that user when setting the limits:

  andrew soft memlock 32768
  andrew hard memlock 524288
Comment 8 Andrew Stitcher 2010-01-28 18:17:31 EST
In para 1:

"that  are much faster than TCP/IP connections"

this isn't necessarily true, how about "that can be faster and have much lower latency than TCP connections"

In para 6:

"the soft limit, which is the initial amount of lockable memory that is allocated"

not really true: it's the initial limit that is assigned, ie the soft limit is the max amount of memory you can lock when you log in unless you change the limit; the hard limit is the maximum you're allowed to set the limit to. How about " which is the initial amount of lockable memory that can be allocated per process"

Note that the limits are per process and not per system. So I'm not sure your recommendation of 16 connections for clients is necessary. However the limit is mainly there to protect the system in case of program error and the memory isn't locked unless it's needed, so there's not much danger in allowing more than you need.

Once you have sized the lockable memory yourself, there is no point in setting a different hard and soft limit. Both are present mainly for interactive use.

[An enhancement request might be to have the rdma module check that amount of lockable memory allowed and try to up the amount if the hard limit allowed and it needed, but it doesn't do anything like that now]

Note You need to log in before you can comment on or make changes to this bug.