Bug 232068

Summary:	ip addresses in cluster.conf do not work
Product:	Red Hat Enterprise Linux 5	Reporter:	Ryan McCabe <rmccabe>
Component:	cman	Assignee:	Christine Caulfield <ccaulfie>
Status:	CLOSED ERRATA	QA Contact:	Cluster QE <mspqa-list>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	5.0	CC:	cluster-maint, jparsons, ldimaggi, pkennedy, rkenna, rmccabe
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	RHBA-2007-0575	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2007-11-07 16:59:18 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	231496
Bug Blocks:	212022

Description Ryan McCabe 2007-03-13 19:59:22 UTC

The old BZ got moved to SELinux and marked modified, so I'm cloning it here.

+++ This bug was initially created as a clone of Bug #231496 +++

Description of problem:
This seems to be a regression from rhel4.
CMAN will not start if there is an ip addr defined as the node name in
cluster.conf, and uname -n returns a host name. 

I suppose if a node was statically configured without a hostname defined, then
uname -n would return the address...and then cman would start; but using ip
addrs worked fine in the past. Have I missed a specification for RHEL5 cluster
suite?  The UI and doc will need to change, if this is 'as designed'.

-- Additional comment from ldimaggi on 2007-03-09 11:21 EST --
Another data point - I tried to create a cluster of Xen guests with fully
qualified hostnames in /etc/hosts on each guest and the guest running luci:

192.168.78.103          March7g1.virt.boston.redhat.com
192.168.78.167          March7g2.virt.boston.redhat.com
192.168.78.164          March7g3.virt.boston.redhat.com
192.168.78.1            March7g4.virt.boston.redhat.com
192.168.78.125          robg4.virt.boston.redhat.com

and on each guest in /etc/sysconfig/network

This created the cluster - but the cluster nodes failed to join the cluster -
/var/log/messages fragment from one of the guests:


Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] AIS Executive Service RELEASE
'subrev 1324 version 0.80.2' 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Copyright (C) 2002-2006
MontaVista Software, Inc and contributors. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] AIS Executive Service: started
and ready to provide service. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Using default multicast address
of 239.192.115.167 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_cpg
loaded. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais cluster closed process group service v1.01' 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_cfg
loaded. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais configuration service' 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_msg
loaded. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais message service B.01.01' 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_lck
loaded. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais distributed locking service B.01.01' 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_evt
loaded. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais event service B.01.01' 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_ckpt
loaded. 
Mar  9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais checkpoint service B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [MAIN ] openais component openais_amf
loaded. 
Mar  9 10:59:02 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais availability management framework B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [MAIN ] openais component openais_clm
loaded. 
Mar  9 10:59:02 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais cluster membership service B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [MAIN ] openais component openais_evs
loaded. 
Mar  9 10:59:02 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais extended virtual synchrony service' 
Mar  9 10:59:02 march7g1 openais[2906]: [MAIN ] openais component openais_cman
loaded. 
Mar  9 10:59:02 march7g1 openais[2906]: [MAIN ] Registering service handler
'openais CMAN membership service 2.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] Token Timeout (10000 ms)
retransmit timeout (495 ms) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] token hold (386 ms) retransmits
before loss (20 retrans) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] join (60 ms) send_join (0 ms)
consensus (4800 ms) merge (200 ms) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] downcheck (1000 ms) fail to recv
const (50 msgs) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] seqno unchanged const (30
rotations) Maximum network MTU 1500 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] window size per rotation (50
messages) maximum messages per rotation (17 messages) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] send threads (0 threads) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] RRP token expired timeout (495 ms) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] RRP token problem counter (2000 ms) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] RRP threshold (10 problem count) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] RRP mode set to none. 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] heartbeat_failures_allowed (0) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] max_network_delay (50 ms) 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] HeartBeat is Disabled. To enable
set heartbeat_failures_allowed > 0 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] Receive multicast socket recv
buffer size (253952 bytes). 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] Transmit multicast socket send
buffer size (253952 bytes). 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] The network interface
[192.168.78.103] is now up. 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] Created or loaded sequence id
0.192.168.78.103 for this ring. 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] entering GATHER state from 15. 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais extended virtual synchrony service' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais cluster membership service B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais availability management framework B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais checkpoint service B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais event service B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais distributed locking service B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais message service B.01.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais configuration service' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais cluster closed process group service v1.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler
'openais CMAN membership service 2.01' 
Mar  9 10:59:02 march7g1 openais[2906]: [CMAN ] CMAN 2.0.60 (built Jan 23 2007
12:42:16) started 
Mar  9 10:59:02 march7g1 openais[2906]: [SYNC ] Not using a virtual synchrony
filter. 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] Creating commit token because I
am the rep. 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] Saving state aru 0 high seq
received 0 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] entering COMMIT state. 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] entering RECOVERY state. 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] position [0] member 192.168.78.103: 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] previous ring seq 0 rep
192.168.78.103 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] aru 0 high delivered 0 received
flag 0 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] Did not need to originate any
messages in recovery. 
Mar  9 10:59:02 march7g1 openais[2906]: [TOTEM] Couldn't store new ring id 4 to
stable storage (Permission denied) 
Mar  9 10:59:02 march7g1 ccsd[2092]: Initial status:: Quorate 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] AIS Executive Service RELEASE
'subrev 1324 version 0.80.2' 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] Copyright (C) 2002-2006
MontaVista Software, Inc and contributors. 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] AIS Executive Service: started
and ready to provide service. 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] Using default multicast address
of 239.192.115.167 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_cpg
loaded. 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais cluster closed process group service v1.01' 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_cfg
loaded. 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais configuration service' 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_msg
loaded. 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais message service B.01.01' 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_lck
loaded. 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais distributed locking service B.01.01' 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_evt
loaded. 
Mar  9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais event service B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_ckpt
loaded. 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais checkpoint service B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_amf
loaded. 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais availability management framework B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_clm
loaded. 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais cluster membership service B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_evs
loaded. 
Mar  9 10:59:04 march7g1 setroubleshoot:      SELinux is preventing the
/usr/sbin/aisexec from using potentially mislabeled files (tmp).      For
complete SELinux messages. run sealert -l 598661a1-7873-4cfb-9d09-116121271959
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais extended virtual synchrony service' 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_cman
loaded. 
Mar  9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler
'openais CMAN membership service 2.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] Token Timeout (10000 ms)
retransmit timeout (495 ms) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] token hold (386 ms) retransmits
before loss (20 retrans) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] join (60 ms) send_join (0 ms)
consensus (4800 ms) merge (200 ms) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] downcheck (1000 ms) fail to recv
const (50 msgs) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] seqno unchanged const (30
rotations) Maximum network MTU 1500 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] window size per rotation (50
messages) maximum messages per rotation (17 messages) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] send threads (0 threads) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] RRP token expired timeout (495 ms) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] RRP token problem counter (2000 ms) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] RRP threshold (10 problem count) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] RRP mode set to none. 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] heartbeat_failures_allowed (0) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] max_network_delay (50 ms) 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] HeartBeat is Disabled. To enable
set heartbeat_failures_allowed > 0 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] Receive multicast socket recv
buffer size (253952 bytes). 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] Transmit multicast socket send
buffer size (253952 bytes). 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] The network interface
[192.168.78.103] is now up. 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] Created or loaded sequence id
0.192.168.78.103 for this ring. 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] entering GATHER state from 15. 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais extended virtual synchrony service' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais cluster membership service B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais availability management framework B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais checkpoint service B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais event service B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais distributed locking service B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais message service B.01.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais configuration service' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais cluster closed process group service v1.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler
'openais CMAN membership service 2.01' 
Mar  9 10:59:04 march7g1 openais[2932]: [CMAN ] CMAN 2.0.60 (built Jan 23 2007
12:42:16) started 
Mar  9 10:59:04 march7g1 openais[2932]: [SYNC ] Not using a virtual synchrony
filter. 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] Creating commit token because I
am the rep. 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] Saving state aru 0 high seq
received 0 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] entering COMMIT state. 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] entering RECOVERY state. 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] position [0] member 192.168.78.103: 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] previous ring seq 0 rep
192.168.78.103 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] aru 0 high delivered 0 received
flag 0 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] Did not need to originate any
messages in recovery. 
Mar  9 10:59:04 march7g1 openais[2932]: [TOTEM] Couldn't store new ring id 4 to
stable storage (Permission denied) 





-- Additional comment from rmccabe on 2007-03-09 11:55 EST --
Are there SELinux AVCs that caused the permission denied?

-- Additional comment from ldimaggi on 2007-03-09 12:07 EST --
Sorry! I missed these:




type=AVC msg=audit(1173460183.091:26): avc:  denied  { write } for  pid=6223
comm="aisexec" name="tmp" dev=dm-0 ino=507393
scontext=system_u:system_r:ccs_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir
type=AVC msg=audit(1173460185.351:27): avc:  denied  { write } for  pid=6255
comm="aisexec" name="tmp" dev=dm-0 ino=507393
scontext=system_u:system_r:ccs_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir

rpm -qa | grep selinux | sort
libselinux-1.33.4-2.el5
libselinux-1.33.4-2.el5
libselinux-python-1.33.4-2.el5
selinux-policy-2.4.6-30.el5
selinux-policy-targeted-2.4.6-30.el5



-- Additional comment from ldimaggi on 2007-03-09 12:20 EST --
From /var/log/messages:

Mar  9 12:20:50 March7g4 setroubleshoot:      SELinux is preventing the
/usr/sbin/aisexec from using potentially mislabeled files
(/tmp/ringid_192.168.78.1).      For complete SELinux messages. run sealert -l
33b6e742-14c5-48a3-b3ac-cd9f257e0e03
Mar  9 12:20:50 March7g4 setroubleshoot:      SELinux is preventing the
/usr/sbin/aisexec from using potentially mislabeled files (ringid_192.168.78.1).
     For complete SELinux messages. run sealert -l
0196bf41-5de4-4864-aa7c-12a04298f371


With SELinux=permissive - the cluster did start

-- Additional comment from dwalsh on 2007-03-09 13:45 EST --
Fixed in selinux-policy-2.4.6-45.el5.src.rpm

Comment 1 Christine Caulfield 2007-03-14 09:29:18 UTC

I just tried this on my RHEL4 system and it doesn't work there either. To be
honest that didn't really surprise me.

Can someone who has this working please attach the cluster.conf and output of
cman_tool join -d

mine is below:

[root@roth-05]# hostname; host `hostname`
roth-05.lab.msp.redhat.com
roth-05.lab.msp.redhat.com has address 10.15.84.105
[root@roth-05]# cman_tool join -d
nodename roth-05.lab.msp.redhat.com not found
nodename roth-05 (truncated) not found
nodename roth-05 doesn't match 10 (10.15.84.105 in cluster.conf)
nodename roth-05 doesn't match roth-06 (roth-06.lab.msp.redhat.com in cluster.conf)
nodename roth-05 doesn't match roth-07 (roth-07.lab.msp.redhat.com in cluster.conf)
nodename roth-05 doesn't match roth-08 (roth-08.lab.msp.redhat.com in cluster.conf)
nodename localhost.localdomain (if lo) not found
nodename localhost (if lo truncated) not found
nodename roth-05.lab.msp.redhat.com (if eth0) not found
nodename roth-05 (if eth0 truncated) not found
cman_tool: local node name "roth-05.lab.msp.redhat.com" not found in cluster.conf

Comment 2 Christine Caulfield 2007-03-15 09:28:18 UTC

Checking in cmanccs.c;
/cvs/cluster/cluster/cman/daemon/cmanccs.c,v  <--  cmanccs.c
new revision: 1.25; previous revision: 1.24
done

I've added support for IPv4 addresses. If anyone needs IPv6 addresses then a new
bugzilla would be appropriate I think.

Comment 3 Steven Dake 2007-03-15 21:01:13 UTC

*** Bug 231496 has been marked as a duplicate of this bug. ***

Comment 4 Kiersten (Kerri) Anderson 2007-04-23 16:52:45 UTC

Fixing Product Name.  Cluster Suite was merged into Red Hat Enterpise Linux for
5.0.  In addition dlm, fence and ccs were merged into the cman package, so
bugzilla should reflect package name where those utilities are located.

Comment 6 RHEL Program Management 2007-05-01 16:20:44 UTC

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 7 Christine Caulfield 2007-05-23 10:31:37 UTC

Oops, I seem to have forgotten to put this on the RHEL5 branch

Checking in cmanccs.c;
/cvs/cluster/cluster/cman/daemon/cmanccs.c,v  <--  cmanccs.c
new revision: 1.21.2.4; previous revision: 1.21.2.3
done

Comment 10 errata-xmlrpc 2007-11-07 16:59:18 UTC

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0575.html