The old BZ got moved to SELinux and marked modified, so I'm cloning it here. +++ This bug was initially created as a clone of Bug #231496 +++ Description of problem: This seems to be a regression from rhel4. CMAN will not start if there is an ip addr defined as the node name in cluster.conf, and uname -n returns a host name. I suppose if a node was statically configured without a hostname defined, then uname -n would return the address...and then cman would start; but using ip addrs worked fine in the past. Have I missed a specification for RHEL5 cluster suite? The UI and doc will need to change, if this is 'as designed'. -- Additional comment from ldimaggi on 2007-03-09 11:21 EST -- Another data point - I tried to create a cluster of Xen guests with fully qualified hostnames in /etc/hosts on each guest and the guest running luci: 192.168.78.103 March7g1.virt.boston.redhat.com 192.168.78.167 March7g2.virt.boston.redhat.com 192.168.78.164 March7g3.virt.boston.redhat.com 192.168.78.1 March7g4.virt.boston.redhat.com 192.168.78.125 robg4.virt.boston.redhat.com and on each guest in /etc/sysconfig/network This created the cluster - but the cluster nodes failed to join the cluster - /var/log/messages fragment from one of the guests: Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2' Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] AIS Executive Service: started and ready to provide service. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Using default multicast address of 239.192.115.167 Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_cpg loaded. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_cfg loaded. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais configuration service' Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_msg loaded. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais message service B.01.01' Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_lck loaded. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_evt loaded. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais event service B.01.01' Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] openais component openais_ckpt loaded. Mar 9 10:59:01 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [MAIN ] openais component openais_amf loaded. Mar 9 10:59:02 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [MAIN ] openais component openais_clm loaded. Mar 9 10:59:02 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [MAIN ] openais component openais_evs loaded. Mar 9 10:59:02 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' Mar 9 10:59:02 march7g1 openais[2906]: [MAIN ] openais component openais_cman loaded. Mar 9 10:59:02 march7g1 openais[2906]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] send threads (0 threads) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] RRP token expired timeout (495 ms) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] RRP token problem counter (2000 ms) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] RRP threshold (10 problem count) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] RRP mode set to none. Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] heartbeat_failures_allowed (0) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] max_network_delay (50 ms) Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] Receive multicast socket recv buffer size (253952 bytes). Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] Transmit multicast socket send buffer size (253952 bytes). Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] The network interface [192.168.78.103] is now up. Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] Created or loaded sequence id 0.192.168.78.103 for this ring. Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] entering GATHER state from 15. Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais event service B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais message service B.01.01' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais configuration service' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' Mar 9 10:59:02 march7g1 openais[2906]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' Mar 9 10:59:02 march7g1 openais[2906]: [CMAN ] CMAN 2.0.60 (built Jan 23 2007 12:42:16) started Mar 9 10:59:02 march7g1 openais[2906]: [SYNC ] Not using a virtual synchrony filter. Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] Creating commit token because I am the rep. Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] Saving state aru 0 high seq received 0 Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] entering COMMIT state. Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] entering RECOVERY state. Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] position [0] member 192.168.78.103: Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] previous ring seq 0 rep 192.168.78.103 Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] aru 0 high delivered 0 received flag 0 Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] Did not need to originate any messages in recovery. Mar 9 10:59:02 march7g1 openais[2906]: [TOTEM] Couldn't store new ring id 4 to stable storage (Permission denied) Mar 9 10:59:02 march7g1 ccsd[2092]: Initial status:: Quorate Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2' Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] AIS Executive Service: started and ready to provide service. Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] Using default multicast address of 239.192.115.167 Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_cpg loaded. Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_cfg loaded. Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais configuration service' Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_msg loaded. Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais message service B.01.01' Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_lck loaded. Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] openais component openais_evt loaded. Mar 9 10:59:03 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais event service B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_ckpt loaded. Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_amf loaded. Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_clm loaded. Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_evs loaded. Mar 9 10:59:04 march7g1 setroubleshoot: SELinux is preventing the /usr/sbin/aisexec from using potentially mislabeled files (tmp). For complete SELinux messages. run sealert -l 598661a1-7873-4cfb-9d09-116121271959 Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] openais component openais_cman loaded. Mar 9 10:59:04 march7g1 openais[2932]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] send threads (0 threads) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] RRP token expired timeout (495 ms) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] RRP token problem counter (2000 ms) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] RRP threshold (10 problem count) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] RRP mode set to none. Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] heartbeat_failures_allowed (0) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] max_network_delay (50 ms) Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] Receive multicast socket recv buffer size (253952 bytes). Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] Transmit multicast socket send buffer size (253952 bytes). Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] The network interface [192.168.78.103] is now up. Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] Created or loaded sequence id 0.192.168.78.103 for this ring. Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] entering GATHER state from 15. Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais event service B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais message service B.01.01' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais configuration service' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' Mar 9 10:59:04 march7g1 openais[2932]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' Mar 9 10:59:04 march7g1 openais[2932]: [CMAN ] CMAN 2.0.60 (built Jan 23 2007 12:42:16) started Mar 9 10:59:04 march7g1 openais[2932]: [SYNC ] Not using a virtual synchrony filter. Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] Creating commit token because I am the rep. Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] Saving state aru 0 high seq received 0 Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] entering COMMIT state. Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] entering RECOVERY state. Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] position [0] member 192.168.78.103: Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] previous ring seq 0 rep 192.168.78.103 Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] aru 0 high delivered 0 received flag 0 Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] Did not need to originate any messages in recovery. Mar 9 10:59:04 march7g1 openais[2932]: [TOTEM] Couldn't store new ring id 4 to stable storage (Permission denied) -- Additional comment from rmccabe on 2007-03-09 11:55 EST -- Are there SELinux AVCs that caused the permission denied? -- Additional comment from ldimaggi on 2007-03-09 12:07 EST -- Sorry! I missed these: type=AVC msg=audit(1173460183.091:26): avc: denied { write } for pid=6223 comm="aisexec" name="tmp" dev=dm-0 ino=507393 scontext=system_u:system_r:ccs_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir type=AVC msg=audit(1173460185.351:27): avc: denied { write } for pid=6255 comm="aisexec" name="tmp" dev=dm-0 ino=507393 scontext=system_u:system_r:ccs_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir rpm -qa | grep selinux | sort libselinux-1.33.4-2.el5 libselinux-1.33.4-2.el5 libselinux-python-1.33.4-2.el5 selinux-policy-2.4.6-30.el5 selinux-policy-targeted-2.4.6-30.el5 -- Additional comment from ldimaggi on 2007-03-09 12:20 EST -- From /var/log/messages: Mar 9 12:20:50 March7g4 setroubleshoot: SELinux is preventing the /usr/sbin/aisexec from using potentially mislabeled files (/tmp/ringid_192.168.78.1). For complete SELinux messages. run sealert -l 33b6e742-14c5-48a3-b3ac-cd9f257e0e03 Mar 9 12:20:50 March7g4 setroubleshoot: SELinux is preventing the /usr/sbin/aisexec from using potentially mislabeled files (ringid_192.168.78.1). For complete SELinux messages. run sealert -l 0196bf41-5de4-4864-aa7c-12a04298f371 With SELinux=permissive - the cluster did start -- Additional comment from dwalsh on 2007-03-09 13:45 EST -- Fixed in selinux-policy-2.4.6-45.el5.src.rpm
I just tried this on my RHEL4 system and it doesn't work there either. To be honest that didn't really surprise me. Can someone who has this working please attach the cluster.conf and output of cman_tool join -d mine is below: [root@roth-05]# hostname; host `hostname` roth-05.lab.msp.redhat.com roth-05.lab.msp.redhat.com has address 10.15.84.105 [root@roth-05]# cman_tool join -d nodename roth-05.lab.msp.redhat.com not found nodename roth-05 (truncated) not found nodename roth-05 doesn't match 10 (10.15.84.105 in cluster.conf) nodename roth-05 doesn't match roth-06 (roth-06.lab.msp.redhat.com in cluster.conf) nodename roth-05 doesn't match roth-07 (roth-07.lab.msp.redhat.com in cluster.conf) nodename roth-05 doesn't match roth-08 (roth-08.lab.msp.redhat.com in cluster.conf) nodename localhost.localdomain (if lo) not found nodename localhost (if lo truncated) not found nodename roth-05.lab.msp.redhat.com (if eth0) not found nodename roth-05 (if eth0 truncated) not found cman_tool: local node name "roth-05.lab.msp.redhat.com" not found in cluster.conf
Checking in cmanccs.c; /cvs/cluster/cluster/cman/daemon/cmanccs.c,v <-- cmanccs.c new revision: 1.25; previous revision: 1.24 done I've added support for IPv4 addresses. If anyone needs IPv6 addresses then a new bugzilla would be appropriate I think.
*** Bug 231496 has been marked as a duplicate of this bug. ***
Fixing Product Name. Cluster Suite was merged into Red Hat Enterpise Linux for 5.0. In addition dlm, fence and ccs were merged into the cman package, so bugzilla should reflect package name where those utilities are located.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Oops, I seem to have forgotten to put this on the RHEL5 branch Checking in cmanccs.c; /cvs/cluster/cluster/cman/daemon/cmanccs.c,v <-- cmanccs.c new revision: 1.21.2.4; previous revision: 1.21.2.3 done
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2007-0575.html