Bug 361761 - Support the active use of multiple network interfaces.
Support the active use of multiple network interfaces.
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: cman (Show other bugs)
All Linux
low Severity low
: ---
: ---
Assigned To: Steven Dake
Cluster QE
: Documentation, FutureFeature
Depends On:
Blocks: 249696 361781
  Show dependency treegraph
Reported: 2007-11-01 09:42 EDT by Scott Crenshaw
Modified: 2016-04-26 09:24 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2008-07-14 11:49:46 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:

Attachments (Terms of Use)

  None (edit)
Description Rob Kenna 2007-11-01 09:42:22 EDT
One way to increase the reliability of cluster interconnect is to use multiple
physical connections.  Currently network bonding can be used.  An alternative is
to support the knowledge of the mulitple interfaces in the cluster
infrastructure layers of openais and cman.

This may well work at the moment but:

1) Must use SCTP (which had serious problems in 5.0).  TCP is currently the
default, though sctp is an unsupported option

2) Conga does not support it's configuration

3) There has been no QE testing

4) There is no documentation.

We should validate that this works, add Conga support, test the solution, and
document it's use.
Comment 2 Christine Caulfield 2007-11-05 09:33:13 EST
It seems to me that dlm_controld, at least, needs some work here as it only
seems to support a single IP address per node. I also reckon that it should
automatically switch over to sctp if more than one address is passed back from cman.

I did a basic test of single-home over sctp and encountered some troubles! I
know this used to work so more investigation is needed.
Comment 3 Christine Caulfield 2007-11-07 08:19:03 EST
OK it looks like my problems were down to insufficient receive memory in the
sockets. dlm_controld (or maybe the startup script?) needs to do something like
the following to make sctp work correctly.

echo 4194304 > /proc/sys/net/core/rmem_default 
echo 4194304 > /proc/sys/net/core/rmem_max     
Comment 4 Christine Caulfield 2007-12-10 11:17:30 EST
Added multi-home code to dlm_controld on CVS head:

Checking in dlm_daemon.h;
/cvs/cluster/cluster/group/dlm_controld/dlm_daemon.h,v  <--  dlm_daemon.h
new revision: 1.14; previous revision: 1.13
Checking in member_cman.c;
/cvs/cluster/cluster/group/dlm_controld/member_cman.c,v  <--  member_cman.c
new revision: 1.9; previous revision: 1.8
Comment 5 Christine Caulfield 2007-12-11 06:26:10 EST
Added the networking params to the init script. NOTE: someone who actually knows
about init scripts might like to check this!

On CVS head:

Checking in cman;
/cvs/cluster/cluster/cman/init.d/cman,v  <--  cman
new revision: 1.33; previous revision: 1.32
Comment 6 Christine Caulfield 2007-12-14 08:03:12 EST
Basic tests of multi-run openais seem to work, but see bz#424681: the default
build restricts you to 2 rings.

Also using "rrp_mode: active" doesn't work if the multicast and port numbers are
the same for all rings. Oddly (to me), "rrp_mode: passive" does work when
configured like this. Changing the multicast address and/or port numbers for
each ring works fine.

With more than one interface configured cman switches to "rrp_mode: active" so
we either need to document this, fix it (and make conga check it) or make cman
use passive.
Comment 7 Rob Kenna 2008-03-20 13:21:15 EDT
Opened for public viewing.
Comment 8 Christine Caulfield 2008-03-31 08:44:07 EDT
Assigning to sdake (sorry steve) so he can check the multi-ring.
Comment 9 Steven Dake 2008-03-31 18:10:33 EDT
I will test it.  One of my switches failed a few months ago so I haven't had a
mulitring setup available to set at all.  So I'll have to work on getting a new

Are you sure your not having both NICs connected to the same interconnect?  That
would explain the port/ring behavior you have in active mode.

In order for redundant ring to work, the interconnects must be completely
seperated, or they must use different port or multicast addresses.

Comment 12 Kiersten (Kerri) Anderson 2008-07-14 11:49:46 EDT
Closing this as not a bug for the cman component.  It is believed this is just a
configuration/setup/management issue and the cluster project wiki has a
description of how to perform this work.  So the work is in the user interfaces
for conga/system-config-cluster and there is already a defect recorded for those

Note You need to log in before you can comment on or make changes to this bug.