Attachment 154966 Details for Bug 240508 – 2 & 3-node cluster.conf files, /var/log/messages on node-1 and node-3

2 & 3-node cluster.conf files, /var/log/messages on node-1 and node-3

logs_and_confs.txt (text/plain), 34.68 KB, created by Marc Curry on 2007-05-18 00:36:45 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Marc Curry

Created: 2007-05-18 00:36:45 UTC

Size: 34.68 KB

patch

obsolete

>In order: 
>     - Original cluster.conf (2node)
>     - Converted cluster.conf (3 nodes)
>     - node-1 /var/log/messages during its cman startup, and add of node-3
>     - node-3 /var/log/messages during its cman startup
>
>
>---------- Original cluster.conf (2node) ---------------------------------------------------
>
><?xml version="1.0"?>
><cluster config_version="1" name="cluster7">
>	<fence_daemon post_fail_delay="0" post_join_delay="3"/>
>	<clusternodes>
>		<clusternode name="node-1.cluster-7.example.com" nodeid="1" votes="1">
>			<fence>
>				<method name="1">
>					<device name="apc7" port="1" switch="1"/>
>				</method>
>			</fence>
>		</clusternode>
>		<clusternode name="node-2.cluster-7.example.com" nodeid="2" votes="1">
>			<fence>
>				<method name="1">
>					<device name="apc7" port="2" switch="1"/>
>				</method>
>			</fence>
>		</clusternode>
>	</clusternodes>
>	<cman expected_votes="1" two_node="1"/>
>	<fencedevices>
>		<fencedevice agent="fence_apc" ipaddr="172.16.36.107" login="nps" name="apc7" passwd="password"/>
>	</fencedevices>
>	<rm>
>		<failoverdomains>
>			<failoverdomain name="prefer1" ordered="1" restricted="1">
>				<failoverdomainnode name="node-1.cluster-7.example.com" priority="1"/>
>				<failoverdomainnode name="node-2.cluster-7.example.com" priority="2"/>
>			</failoverdomain>
>		</failoverdomains>
>		<resources>
>			<fs device="/dev/sdb1" force_fsck="0" force_unmount="0" fstype="ext3" mountpoint="/web" name="docroot" options="" self_fence="0"/>
>			<ip address="172.16.36.207" monitor_link="1"/>
>			<script file="/etc/rc.d/init.d/httpd" name="httpd"/>
>		</resources>
>		<service autostart="1" domain="prefer1" name="webby" recovery="relocate">
>			<fs ref="docroot"/>
>			<ip ref="172.16.36.207"/>
>			<script ref="httpd"/>
>		</service>
>	</rm>
></cluster>
>
>---------- Converted cluster.conf (3 nodes) ------------------------------------------------
>
><?xml version="1.0"?>
><cluster config_version="2" name="cluster7">
>	<fence_daemon post_fail_delay="0" post_join_delay="3"/>
>	<clusternodes>
>		<clusternode name="node-1.cluster-7.example.com" nodeid="1" votes="1">
>			<fence>
>				<method name="1">
>					<device name="apc7" port="1" switch="1"/>
>				</method>
>			</fence>
>		</clusternode>
>		<clusternode name="node-2.cluster-7.example.com" nodeid="2" votes="1">
>			<fence>
>				<method name="1">
>					<device name="apc7" port="2" switch="1"/>
>				</method>
>			</fence>
>		</clusternode>
>		<clusternode name="node-3.cluster-7.example.com" nodeid="3" votes="1">
>			<fence>
>				<method name="1">
>					<device name="apc7" port="3" switch="1"/>
>				</method>
>			</fence>
>		</clusternode>
>	</clusternodes>
>	<cman/>
>	<fencedevices>
>		<fencedevice agent="fence_apc" ipaddr="172.16.36.107" login="nps" name="apc7" passwd="password"/>
>	</fencedevices>
>	<rm>
>		<failoverdomains>
>			<failoverdomain name="prefer1" ordered="1" restricted="1">
>				<failoverdomainnode name="node-1.cluster-7.example.com" priority="1"/>
>				<failoverdomainnode name="node-2.cluster-7.example.com" priority="2"/>
>				<failoverdomainnode name="node-3.cluster-7.example.com" priority="2"/>
>			</failoverdomain>
>		</failoverdomains>
>		<resources>
>			<fs device="/dev/sdb1" force_fsck="0" force_unmount="0" fstype="ext3" mountpoint="/web" name="docroot" options="" self_fence="0"/>
>			<ip address="172.16.36.207" monitor_link="1"/>
>			<script file="/etc/rc.d/init.d/httpd" name="httpd"/>
>		</resources>
>		<service autostart="1" domain="prefer1" name="webby" recovery="relocate">
>			<fs ref="docroot"/>
>			<ip ref="172.16.36.207"/>
>			<script ref="httpd"/>
>		</service>
>	</rm>
></cluster>
>
>---------- node-1 /var/log/messages during its cman startup, and add of node-3 -------------
>
>May 17 19:15:34 node-1 kernel: SELinux: initialized (dev configfs, type configfs), uses genfs_contexts
>May 17 19:15:34 node-1 kernel: DLM (built Jan 26 2007 14:15:47) installed
>May 17 19:15:34 node-1 kernel: GFS2 (built Jan 26 2007 14:16:08) installed
>May 17 19:15:34 node-1 kernel: Lock_DLM (built Jan 26 2007 14:16:14) installed
>May 17 19:15:34 node-1 ccsd[20993]: Starting ccsd 2.0.60: 
>May 17 19:15:34 node-1 ccsd[20993]:  Built: Jan 23 2007 12:42:25 
>May 17 19:15:34 node-1 ccsd[20993]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved. 
>May 17 19:15:34 node-1 ccsd[20993]: cluster.conf (cluster name = cluster7, version = 1) found. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] AIS Executive Service: started and ready to provide service. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Using default multicast address of 239.192.104.8 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_cpg loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_cfg loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais configuration service' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_msg loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais message service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_lck loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_evt loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais event service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_ckpt loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_amf loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_clm loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_evs loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] openais component openais_cman loaded. 
>May 17 19:15:37 node-1 openais[21000]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] send threads (0 threads) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] RRP token expired timeout (495 ms) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] RRP token problem counter (2000 ms) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] RRP threshold (10 problem count) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] RRP mode set to none. 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] heartbeat_failures_allowed (0) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] max_network_delay (50 ms) 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] The network interface [172.16.36.71] is now up. 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] Created or loaded sequence id 0.172.16.36.71 for this ring. 
>May 17 19:15:37 node-1 openais[21000]: [TOTEM] entering GATHER state from 15. 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais event service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais message service B.01.01' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais configuration service' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' 
>May 17 19:15:37 node-1 openais[21000]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' 
>May 17 19:15:37 node-1 openais[21000]: [CMAN ] CMAN 2.0.60 (built Jan 23 2007 12:42:29) started 
>May 17 19:15:37 node-1 ccsd[20993]: Initial status:: Quorate 
>May 17 19:15:41 node-1 openais[21000]: [SYNC ] Not using a virtual synchrony filter. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Creating commit token because I am the rep. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Saving state aru 0 high seq received 0 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] entering COMMIT state. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] entering RECOVERY state. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] position [0] member 172.16.36.71: 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] previous ring seq 0 rep 172.16.36.71 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] aru 0 high delivered 0 received flag 0 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Did not need to originate any messages in recovery. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Storing new sequence id for ring 4 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Sending initial ORF token 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] New Configuration: 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] Members Left: 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] Members Joined: 
>May 17 19:15:41 node-1 openais[21000]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] New Configuration: 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.71)  
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] Members Left: 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] Members Joined: 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.71)  
>May 17 19:15:41 node-1 openais[21000]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] entering OPERATIONAL state. 
>May 17 19:15:41 node-1 openais[21000]: [CMAN ] quorum regained, resuming activity 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] got nodejoin message 172.16.36.71 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] entering GATHER state from 11. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Creating commit token because I am the rep. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Saving state aru 9 high seq received 9 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] entering COMMIT state. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] entering RECOVERY state. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] position [0] member 172.16.36.71: 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] previous ring seq 4 rep 172.16.36.71 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] aru 9 high delivered 9 received flag 0 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] position [1] member 172.16.36.72: 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] previous ring seq 0 rep 172.16.36.72 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] aru 0 high delivered 0 received flag 0 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Did not need to originate any messages in recovery. 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Storing new sequence id for ring 8 
>May 17 19:15:41 node-1 openais[21000]: [TOTEM] Sending initial ORF token 
>May 17 19:15:41 node-1 openais[21000]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] New Configuration: 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.71)  
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] Members Left: 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] Members Joined: 
>May 17 19:15:45 node-1 openais[21000]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] New Configuration: 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.71)  
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.72)  
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] Members Left: 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] Members Joined: 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.72)  
>May 17 19:15:45 node-1 openais[21000]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:15:45 node-1 openais[21000]: [TOTEM] entering OPERATIONAL state. 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] got nodejoin message 172.16.36.71 
>May 17 19:15:45 node-1 openais[21000]: [CLM  ] got nodejoin message 172.16.36.72 
>May 17 19:17:13 node-1 kernel: dlm: connecting to 2
>May 17 19:17:13 node-1 kernel: dlm: got connection from 2
>May 17 19:17:22 node-1 clurgmgrd[21160]: <notice> Resource Group Manager Starting 
>May 17 19:17:22 node-1 clurgmgrd: [21160]: <info> Executing /etc/rc.d/init.d/httpd stop 
>May 17 19:17:23 node-1 clurgmgrd: [21160]: <info> /dev/sdb1 is not mounted 
>May 17 19:17:35 node-1 clurgmgrd[21160]: <notice> Starting stopped service service:webby 
>May 17 19:17:36 node-1 clurgmgrd: [21160]: <info> mounting /dev/sdb1 on /web 
>May 17 19:17:36 node-1 kernel: kjournald starting.  Commit interval 5 seconds
>May 17 19:17:36 node-1 kernel: EXT3 FS on sdb1, internal journal
>May 17 19:17:36 node-1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
>May 17 19:17:36 node-1 kernel: SELinux: initialized (dev sdb1, type ext3), uses xattr
>May 17 19:17:36 node-1 clurgmgrd: [21160]: <info> Adding IPv4 address 172.16.36.207 to eth0 
>May 17 19:17:36 node-1 avahi-daemon[2031]: Registering new address record for 172.16.36.207 on eth0.
>May 17 19:17:37 node-1 in.rdiscd[21602]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use
>May 17 19:17:37 node-1 in.rdiscd[21602]: Failed joining addresses 
>May 17 19:17:37 node-1 clurgmgrd: [21160]: <info> Executing /etc/rc.d/init.d/httpd start 
>May 17 19:17:37 node-1 clurgmgrd[21160]: <notice> Service service:webby started 
>May 17 19:17:45 node-1 clurgmgrd: [21160]: <info> Executing /etc/rc.d/init.d/httpd status 
>May 17 19:18:45 node-1 last message repeated 2 times
>May 17 19:20:15 node-1 last message repeated 3 times
>May 17 19:21:15 node-1 last message repeated 2 times
>May 17 19:21:16 node-1 clurgmgrd[21160]: <notice> Stopping service service:webby 
>May 17 19:21:16 node-1 clurgmgrd: [21160]: <info> Executing /etc/rc.d/init.d/httpd stop 
>May 17 19:21:16 node-1 clurgmgrd: [21160]: <info> Removing IPv4 address 172.16.36.207 from eth0 
>May 17 19:21:16 node-1 avahi-daemon[2031]: Withdrawing address record for 172.16.36.207 on eth0.
>May 17 19:21:26 node-1 clurgmgrd: [21160]: <info> unmounting /web 
>May 17 19:21:26 node-1 clurgmgrd[21160]: <notice> Service service:webby is stopped 
>May 17 19:28:44 node-1 gconfd (root-25082): starting (version 2.14.0), pid 25082 user 'root'
>May 17 19:28:44 node-1 gconfd (root-25082): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
>May 17 19:28:44 node-1 gconfd (root-25082): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 1
>May 17 19:28:44 node-1 gconfd (root-25082): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
>May 17 19:29:14 node-1 gconfd (root-25082): GConf server is not in use, shutting down.
>May 17 19:29:14 node-1 gconfd (root-25082): Exiting
>May 17 19:29:21 node-1 gconfd (root-25138): starting (version 2.14.0), pid 25138 user 'root'
>May 17 19:29:21 node-1 gconfd (root-25138): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
>May 17 19:29:21 node-1 gconfd (root-25138): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 1
>May 17 19:29:21 node-1 gconfd (root-25138): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
>May 17 19:31:42 node-1 ccsd[20993]: Update of cluster.conf complete (version 1 -> 2). 
>May 17 19:31:45 node-1 clurgmgrd[21160]: <notice> Reconfiguring 
>May 17 19:35:10 node-1 openais[21000]: [TOTEM] entering GATHER state from 11. 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] Creating commit token because I am the rep. 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] Saving state aru 77 high seq received 77 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] entering COMMIT state. 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] entering RECOVERY state. 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] position [0] member 172.16.36.71: 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] previous ring seq 8 rep 172.16.36.71 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] aru 77 high delivered 77 received flag 0 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] position [1] member 172.16.36.72: 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] previous ring seq 8 rep 172.16.36.71 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] aru 77 high delivered 77 received flag 0 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] position [2] member 172.16.36.73: 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] previous ring seq 4 rep 172.16.36.73 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] aru a high delivered a received flag 0 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] Did not need to originate any messages in recovery. 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] Storing new sequence id for ring c 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] Sending initial ORF token 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] New Configuration: 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.71)  
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.72)  
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] Members Left: 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] Members Joined: 
>May 17 19:35:11 node-1 openais[21000]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] New Configuration: 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.71)  
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.72)  
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.73)  
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] Members Left: 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] Members Joined: 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] 	r(0) ip(172.16.36.73)  
>May 17 19:35:11 node-1 openais[21000]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:35:11 node-1 openais[21000]: [TOTEM] entering OPERATIONAL state. 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] got nodejoin message 172.16.36.71 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] got nodejoin message 172.16.36.72 
>May 17 19:35:11 node-1 openais[21000]: [CLM  ] got nodejoin message 172.16.36.73 
>May 17 19:35:11 node-1 openais[21000]: [CPG  ] got joinlist message from node 3 
>May 17 19:35:11 node-1 openais[21000]: [CPG  ] got joinlist message from node 1 
>May 17 19:35:11 node-1 openais[21000]: [CPG  ] got joinlist message from node 2 
>May 17 19:35:18 node-1 kernel: dlm: connecting to 3
>May 17 19:35:18 node-1 kernel: dlm: got connection from 3
>
>---------- node-3 /var/log/messages during its cman startup --------------------------------
>
>May 17 19:35:10 node-3 kernel: SELinux: initialized (dev configfs, type configfs), uses genfs_contexts
>May 17 19:35:10 node-3 kernel: DLM (built Jan 26 2007 14:15:47) installed
>May 17 19:35:10 node-3 kernel: GFS2 (built Jan 26 2007 14:16:08) installed
>May 17 19:35:10 node-3 kernel: Lock_DLM (built Jan 26 2007 14:16:14) installed
>May 17 19:35:10 node-3 ccsd[15071]: Starting ccsd 2.0.60: 
>May 17 19:35:10 node-3 ccsd[15071]:  Built: Jan 23 2007 12:42:25 
>May 17 19:35:10 node-3 ccsd[15071]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved. 
>May 17 19:35:10 node-3 ccsd[15071]: cluster.conf (cluster name = cluster7, version = 2) found. 
>May 17 19:35:10 node-3 ccsd[15071]: Remote copy of cluster.conf is from quorate node. 
>May 17 19:35:10 node-3 ccsd[15071]:  Local version # : 2 
>May 17 19:35:10 node-3 ccsd[15071]:  Remote version #: 2 
>May 17 19:35:10 node-3 ccsd[15071]: Remote copy of cluster.conf is from quorate node. 
>May 17 19:35:10 node-3 ccsd[15071]:  Local version # : 2 
>May 17 19:35:10 node-3 ccsd[15071]:  Remote version #: 2 
>May 17 19:35:10 node-3 ccsd[15071]: Remote copy of cluster.conf is from quorate node. 
>May 17 19:35:10 node-3 ccsd[15071]:  Local version # : 2 
>May 17 19:35:10 node-3 ccsd[15071]:  Remote version #: 2 
>May 17 19:35:10 node-3 ccsd[15071]: Remote copy of cluster.conf is from quorate node. 
>May 17 19:35:10 node-3 ccsd[15071]:  Local version # : 2 
>May 17 19:35:10 node-3 ccsd[15071]:  Remote version #: 2 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] AIS Executive Service: started and ready to provide service. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Using default multicast address of 239.192.104.8 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_cpg loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_cfg loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais configuration service' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_msg loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais message service B.01.01' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_lck loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_evt loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais event service B.01.01' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_ckpt loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_amf loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_clm loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_evs loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] openais component openais_cman loaded. 
>May 17 19:35:10 node-3 openais[15077]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' 
>May 17 19:35:10 node-3 openais[15077]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
>May 17 19:35:10 node-3 openais[15077]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] send threads (0 threads) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] RRP token expired timeout (495 ms) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] RRP token problem counter (2000 ms) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] RRP threshold (10 problem count) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] RRP mode set to none. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] heartbeat_failures_allowed (0) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] max_network_delay (50 ms) 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] The network interface [172.16.36.73] is now up. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Created or loaded sequence id 0.172.16.36.73 for this ring. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] entering GATHER state from 15. 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais event service B.01.01' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais message service B.01.01' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais configuration service' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' 
>May 17 19:35:11 node-3 openais[15077]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' 
>May 17 19:35:11 node-3 openais[15077]: [CMAN ] CMAN 2.0.60 (built Jan 23 2007 12:42:29) started 
>May 17 19:35:11 node-3 openais[15077]: [SYNC ] Not using a virtual synchrony filter. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Creating commit token because I am the rep. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Saving state aru 0 high seq received 0 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] entering COMMIT state. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] entering RECOVERY state. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] position [0] member 172.16.36.73: 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] previous ring seq 0 rep 172.16.36.73 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] aru 0 high delivered 0 received flag 0 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Did not need to originate any messages in recovery. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Storing new sequence id for ring 4 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Sending initial ORF token 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] New Configuration: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] Members Left: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] Members Joined: 
>May 17 19:35:11 node-3 openais[15077]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] New Configuration: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] 	r(0) ip(172.16.36.73)  
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] Members Left: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] Members Joined: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] 	r(0) ip(172.16.36.73)  
>May 17 19:35:11 node-3 openais[15077]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] entering OPERATIONAL state. 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] got nodejoin message 172.16.36.73 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] entering GATHER state from 11. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Saving state aru a high seq received a 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] entering COMMIT state. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] entering RECOVERY state. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] position [0] member 172.16.36.71: 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] previous ring seq 8 rep 172.16.36.71 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] aru 77 high delivered 77 received flag 0 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] position [1] member 172.16.36.72: 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] previous ring seq 8 rep 172.16.36.71 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] aru 77 high delivered 77 received flag 0 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] position [2] member 172.16.36.73: 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] previous ring seq 4 rep 172.16.36.73 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] aru a high delivered a received flag 0 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Did not need to originate any messages in recovery. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] Storing new sequence id for ring c 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] New Configuration: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] 	r(0) ip(172.16.36.73)  
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] Members Left: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] Members Joined: 
>May 17 19:35:11 node-3 openais[15077]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] CLM CONFIGURATION CHANGE 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] New Configuration: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] 	r(0) ip(172.16.36.71)  
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] 	r(0) ip(172.16.36.72)  
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] 	r(0) ip(172.16.36.73)  
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] Members Left: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] Members Joined: 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] 	r(0) ip(172.16.36.71)  
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] 	r(0) ip(172.16.36.72)  
>May 17 19:35:11 node-3 openais[15077]: [SYNC ] This node is within the primary component and will provide service. 
>May 17 19:35:11 node-3 openais[15077]: [TOTEM] entering OPERATIONAL state. 
>May 17 19:35:11 node-3 openais[15077]: [CMAN ] quorum regained, resuming activity 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] got nodejoin message 172.16.36.71 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] got nodejoin message 172.16.36.72 
>May 17 19:35:11 node-3 openais[15077]: [CLM  ] got nodejoin message 172.16.36.73 
>May 17 19:35:11 node-3 openais[15077]: [CPG  ] got joinlist message from node 3 
>May 17 19:35:11 node-3 openais[15077]: [CPG  ] got joinlist message from node 1 
>May 17 19:35:11 node-3 openais[15077]: [CPG  ] got joinlist message from node 2 
>May 17 19:35:11 node-3 ccsd[15071]: Initial status:: Quorate 
>May 17 19:35:18 node-3 kernel: dlm: connecting to 1
>May 17 19:35:18 node-3 kernel: dlm: got connection from 2
>May 17 19:35:18 node-3 kernel: dlm: got connection from 1
>May 17 19:35:27 node-3 clurgmgrd[15133]: <notice> Resource Group Manager Starting 
>May 17 19:35:27 node-3 clurgmgrd: [15133]: <info> Executing /etc/rc.d/init.d/httpd stop 
>May 17 19:35:27 node-3 clurgmgrd: [15133]: <info> /dev/sdb1 is not mounted 
>May 17 19:36:03 node-3 dhclient: DHCPREQUEST on eth0 to 172.16.0.1 port 67
>May 17 19:36:03 node-3 dhclient: DHCPACK from 172.16.0.1
>May 17 19:36:03 node-3 dhclient: bound to 172.16.36.73 -- renewal in 9690 seconds.
>May 17 19:40:02 node-3 clurgmgrd[15133]: <notice> Starting stopped service service:webby 
>May 17 19:40:02 node-3 clurgmgrd: [15133]: <info> mounting /dev/sdb1 on /web 
>May 17 19:40:02 node-3 kernel: kjournald starting.  Commit interval 5 seconds
>May 17 19:40:02 node-3 kernel: EXT3 FS on sdb1, internal journal
>May 17 19:40:02 node-3 kernel: EXT3-fs: mounted filesystem with ordered data mode.
>May 17 19:40:02 node-3 kernel: SELinux: initialized (dev sdb1, type ext3), uses xattr
>May 17 19:40:02 node-3 clurgmgrd: [15133]: <info> Adding IPv4 address 172.16.36.207 to eth0 
>May 17 19:40:02 node-3 avahi-daemon[2032]: Registering new address record for 172.16.36.207 on eth0.
>May 17 19:40:03 node-3 in.rdiscd[15604]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use
>May 17 19:40:03 node-3 in.rdiscd[15604]: Failed joining addresses 
>May 17 19:40:03 node-3 clurgmgrd: [15133]: <info> Executing /etc/rc.d/init.d/httpd start 
>May 17 19:40:03 node-3 clurgmgrd[15133]: <notice> Service service:webby started 
>May 17 19:40:10 node-3 clurgmgrd: [15133]: <info> Executing /etc/rc.d/init.d/httpd status 
>
>----------- END ----------------------------------------------------------------------------

Actions: View

Attachments on bug 240508: 154966 | 154984