Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 314735 Details for
Bug 459731
Fenced node rejoins the cluster and fails abruptly without reason
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Contains /var/log/messages files, cluster configuration & automation script
upload.txt (text/plain), 77.40 KB, created by
Rajeev
on 2008-08-21 17:35:47 UTC
(
hide
)
Description:
Contains /var/log/messages files, cluster configuration & automation script
Filename:
MIME Type:
Creator:
Rajeev
Created:
2008-08-21 17:35:47 UTC
Size:
77.40 KB
patch
obsolete
> >-------- cluster.conf -------- > ><?xml version="1.0"?> ><cluster config_version="2" name="sgrhel5"> > <totem token="28000"/> > <fence_daemon post_fail_delay="0" post_join_delay="30"/> > <clusternodes> > <clusternode name="rum.ind.hp.com" nodeid="1" votes="1"> > <fence> > <method name="1"> > <device name="fencerum"/> > </method> > </fence> > </clusternode> > <clusternode name="vodka.ind.hp.com" nodeid="2" votes="1"> > <fence> > <method name="1"> > <device name="fencevodka"/> > </method> > </fence> > </clusternode> > <clusternode name="beer.ind.hp.com" nodeid="3" votes="1"> > <fence> > <method name="1"> > <device name="fencebeer"/> > </method> > </fence> > </clusternode> > </clusternodes> > <cman/> > <fencedevices> > <fencedevice agent="fence_ilo" hostname="vodka-ilo" login="root" name="fencevodka" passwd="lvmtest"/> > <fencedevice agent="fence_ilo" hostname="rum-ilo" login="root" name="fencerum" passwd="lvmtest"/> > <fencedevice agent="fence_ilo" hostname="beer-ilo" login="root" name="fencebeer" passwd="lvmtest"/> > </fencedevices> > <rm> > <failoverdomains/> > <resources/> > </rm> ></cluster> > >----- /var/log/messages (rum) ----- > >Aug 17 23:27:55 rum openais[2672]: [TOTEM] entering GATHER state from 11. >Aug 17 23:27:55 rum openais[2672]: [TOTEM] Saving state aru 23 high seq received 23 >Aug 17 23:27:55 rum openais[2672]: [TOTEM] Storing new sequence id for ring 4dac >Aug 17 23:27:55 rum openais[2672]: [TOTEM] entering COMMIT state. >Aug 17 23:27:55 rum openais[2672]: [TOTEM] entering RECOVERY state. >Aug 17 23:27:55 rum openais[2672]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:27:55 rum openais[2672]: [TOTEM] previous ring seq 19880 rep 16.181.160.138 >Aug 17 23:27:55 rum openais[2672]: [TOTEM] aru b high delivered b received flag 1 >Aug 17 23:27:55 rum openais[2672]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:27:55 rum openais[2672]: [TOTEM] previous ring seq 19880 rep 16.181.160.140 >Aug 17 23:27:55 rum openais[2672]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:27:55 rum openais[2672]: [TOTEM] position [2] member 16.181.160.141: >Aug 17 23:27:55 rum openais[2672]: [TOTEM] previous ring seq 19880 rep 16.181.160.140 >Aug 17 23:27:55 rum openais[2672]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:27:55 rum openais[2672]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:27:55 rum openais[2672]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:27:55 rum openais[2672]: [CLM ] New Configuration: >Aug 17 23:27:55 rum openais[2672]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:27:55 rum openais[2672]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:27:55 rum openais[2672]: [CLM ] Members Left: >Aug 17 23:27:55 rum openais[2672]: [CLM ] Members Joined: >Aug 17 23:27:55 rum openais[2672]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:27:55 rum openais[2672]: [CLM ] New Configuration: >Aug 17 23:27:55 rum openais[2672]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:27:55 rum openais[2672]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:27:55 rum openais[2672]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:27:55 rum openais[2672]: [CLM ] Members Left: >Aug 17 23:27:55 rum openais[2672]: [CLM ] Members Joined: >Aug 17 23:27:55 rum openais[2672]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:27:55 rum openais[2672]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:27:55 rum openais[2672]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:27:55 rum openais[2672]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:27:55 rum openais[2672]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:27:55 rum openais[2672]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:27:55 rum openais[2672]: [CPG ] got joinlist message from node 2 >Aug 17 23:27:55 rum openais[2672]: [CPG ] got joinlist message from node 1 >Aug 17 23:27:55 rum openais[2672]: [CPG ] got joinlist message from node 3 >Aug 17 23:28:00 rum kernel: dlm: connecting to 3 >Aug 17 23:28:00 rum kernel: dlm: got connection from 3 >Aug 17 23:31:18 rum avahi-daemon[3312]: Interface eth3.IPv6 no longer relevant for mDNS. >Aug 17 23:31:18 rum avahi-daemon[3312]: Leaving mDNS multicast group on interface eth3.IPv6 with address fe80::214:c2ff:fe58:6a8. >Aug 17 23:31:18 rum avahi-daemon[3312]: Interface eth3.IPv4 no longer relevant for mDNS. >Aug 17 23:31:18 rum avahi-daemon[3312]: Leaving mDNS multicast group on interface eth3.IPv4 with address 16.181.160.141. >Aug 17 23:31:18 rum avahi-daemon[3312]: Withdrawing address record for fe80::214:c2ff:fe58:6a8 on eth3. >Aug 17 23:31:18 rum avahi-daemon[3312]: Withdrawing address record for 16.181.160.141 on eth3. >Aug 17 23:31:35 rum ntpd[3069]: sendto(66.96.98.9) (fd=22): Network is unreachable >Aug 17 23:31:40 rum ntpd[3069]: sendto(69.36.240.252) (fd=22): Network is unreachable >Aug 17 23:31:43 rum ntpd[3069]: sendto(66.79.167.34) (fd=22): Network is unreachable >Aug 17 23:31:45 rum openais[2672]: [TOTEM] The token was lost in the OPERATIONAL state. >Aug 17 23:31:45 rum openais[2672]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). >Aug 17 23:31:45 rum openais[2672]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). >Aug 17 23:31:45 rum openais[2672]: [TOTEM] The network interface is down. >Aug 17 23:31:45 rum openais[2672]: [TOTEM] entering GATHER state from 15. >Aug 17 23:31:45 rum openais[2672]: [TOTEM] entering GATHER state from 2. >Aug 17 23:31:50 rum openais[2672]: [TOTEM] entering GATHER state from 0. >Aug 17 23:31:50 rum openais[2672]: [TOTEM] Creating commit token because I am the rep. >Aug 17 23:31:50 rum openais[2672]: [TOTEM] Saving state aru 48 high seq received 48 >Aug 17 23:31:50 rum openais[2672]: [TOTEM] Storing new sequence id for ring 4db0 >Aug 17 23:31:50 rum openais[2672]: [TOTEM] entering COMMIT state. >Aug 17 23:31:50 rum openais[2672]: [TOTEM] entering RECOVERY state. >Aug 17 23:31:50 rum openais[2672]: [TOTEM] position [0] member 127.0.0.1: >Aug 17 23:31:50 rum openais[2672]: [TOTEM] previous ring seq 19884 rep 16.181.160.138 >Aug 17 23:31:50 rum openais[2672]: [TOTEM] aru 48 high delivered 48 received flag 1 >Aug 17 23:31:50 rum openais[2672]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:31:50 rum openais[2672]: [TOTEM] Sending initial ORF token >Aug 17 23:31:50 rum openais[2672]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:31:50 rum kernel: dlm: closing connection to node 1 >Aug 17 23:31:50 rum openais[2672]: [CLM ] New Configuration: >Aug 17 23:31:50 rum kernel: dlm: closing connection to node 2 >Aug 17 23:31:50 rum openais[2672]: [CLM ] r(0) ip(127.0.0.1) >Aug 17 23:31:50 rum openais[2672]: [CLM ] Members Left: >Aug 17 23:31:50 rum openais[2672]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:31:50 rum openais[2672]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:31:50 rum openais[2672]: [CLM ] Members Joined: >Aug 17 23:31:50 rum openais[2672]: [CMAN ] quorum lost, blocking activity >Aug 17 23:31:50 rum openais[2672]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:31:50 rum openais[2672]: [CLM ] New Configuration: >Aug 17 23:31:50 rum openais[2672]: [CLM ] r(0) ip(127.0.0.1) >Aug 17 23:31:50 rum openais[2672]: [CLM ] Members Left: >Aug 17 23:31:50 rum openais[2672]: [CLM ] Members Joined: >Aug 17 23:31:50 rum openais[2672]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:31:50 rum openais[2672]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:31:50 rum openais[2672]: [MAIN ] Node rum.ind.hp.com not joined to cman because it has rejoined an inquorate cluster >Aug 17 23:31:50 rum openais[2672]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:31:50 rum openais[2672]: [EVT ] recovery error node: r(0) ip(127.0.0.1) not found >Aug 17 23:31:50 rum dlm_controld[2696]: groupd is down, exiting >Aug 17 23:31:50 rum gfs_controld[2702]: groupd_dispatch error -1 errno 11 >Aug 17 23:31:50 rum gfs_controld[2702]: groupd connection died >Aug 17 23:31:50 rum gfs_controld[2702]: cluster is down, exiting >Aug 17 23:31:50 rum kernel: dlm: closing connection to node 3 >Aug 17 23:32:09 rum shutdown[4974]: shutting down for system halt >Aug 17 23:32:10 rum modclusterd: shutdown succeeded >Aug 17 23:34:53 rum syslogd 1.4.1: restart. >Aug 17 23:34:53 rum kernel: klogd 1.4.1, log source = /proc/kmsg started. > >............. >............ > >Aug 17 23:34:54 rum rpc.statd[2553]: Version 1.0.9 Starting >Aug 17 23:34:54 rum rpc.statd[2553]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user >Aug 17 23:34:54 rum kernel: DLM (built Oct 10 2007 16:34:47) installed >Aug 17 23:34:55 rum kernel: GFS2 (built Oct 10 2007 16:35:30) installed >Aug 17 23:34:55 rum kernel: Lock_DLM (built Oct 10 2007 16:35:42) installed >Aug 17 23:34:55 rum ccsd[2655]: Starting ccsd 2.0.73: >Aug 17 23:34:55 rum ccsd[2655]: Built: Sep 19 2007 16:04:06 >Aug 17 23:34:55 rum ccsd[2655]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. >Aug 17 23:34:55 rum ccsd[2655]: cluster.conf (cluster name = sgrhel5, version = 2) found. >Aug 17 23:34:55 rum ccsd[2655]: Remote copy of cluster.conf is from quorate node. >Aug 17 23:34:55 rum ccsd[2655]: Local version # : 2 >Aug 17 23:34:55 rum ccsd[2655]: Remote version #: 2 >Aug 17 23:34:56 rum ccsd[2655]: Remote copy of cluster.conf is from quorate node. >Aug 17 23:34:56 rum ccsd[2655]: Local version # : 2 >Aug 17 23:34:56 rum ccsd[2655]: Remote version #: 2 >Aug 17 23:34:56 rum ccsd[2655]: Remote copy of cluster.conf is from quorate node. >Aug 17 23:34:56 rum ccsd[2655]: Local version # : 2 >Aug 17 23:34:56 rum ccsd[2655]: Remote version #: 2 >Aug 17 23:34:56 rum ccsd[2655]: Remote copy of cluster.conf is from quorate node. >Aug 17 23:34:56 rum ccsd[2655]: Local version # : 2 >Aug 17 23:34:56 rum ccsd[2655]: Remote version #: 2 >Aug 17 23:34:56 rum openais[2663]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] AIS Executive Service: started and ready to provide service. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Using default multicast address of 239.192.54.215 >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_cpg loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_cfg loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais configuration service' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_msg loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais message service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_lck loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_evt loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais event service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_ckpt loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_amf loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_clm loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_evs loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' >Aug 17 23:34:56 rum openais[2663]: [MAIN ] openais component openais_cman loaded. >Aug 17 23:34:56 rum openais[2663]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Token Timeout (28000 ms) retransmit timeout (1386 ms) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] token hold (1098 ms) retransmits before loss (20 retrans) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 >Aug 17 23:34:56 rum openais[2663]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] send threads (0 threads) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] RRP token expired timeout (1386 ms) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] RRP token problem counter (2000 ms) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] RRP threshold (10 problem count) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] RRP mode set to none. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] heartbeat_failures_allowed (0) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] max_network_delay (50 ms) >Aug 17 23:34:56 rum openais[2663]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). >Aug 17 23:34:56 rum openais[2663]: [TOTEM] The network interface [16.181.160.141] is now up. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Created or loaded sequence id 19884.16.181.160.141 for this ring. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] entering GATHER state from 15. >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais event service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais message service B.01.01' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais configuration service' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' >Aug 17 23:34:56 rum openais[2663]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' >Aug 17 23:34:56 rum openais[2663]: [CMAN ] CMAN 2.0.73 (built Sep 19 2007 16:04:10) started >Aug 17 23:34:56 rum openais[2663]: [SYNC ] Not using a virtual synchrony filter. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Creating commit token because I am the rep. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Saving state aru 0 high seq received 0 >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Storing new sequence id for ring 4db0 >Aug 17 23:34:56 rum openais[2663]: [TOTEM] entering COMMIT state. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] entering RECOVERY state. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] position [0] member 16.181.160.141: >Aug 17 23:34:56 rum openais[2663]: [TOTEM] previous ring seq 19884 rep 16.181.160.141 >Aug 17 23:34:56 rum openais[2663]: [TOTEM] aru 0 high delivered 0 received flag 1 >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] Sending initial ORF token >Aug 17 23:34:56 rum openais[2663]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:34:56 rum openais[2663]: [CLM ] New Configuration: >Aug 17 23:34:56 rum openais[2663]: [CLM ] Members Left: >Aug 17 23:34:56 rum openais[2663]: [CLM ] Members Joined: >Aug 17 23:34:56 rum openais[2663]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:34:56 rum openais[2663]: [CLM ] New Configuration: >Aug 17 23:34:56 rum openais[2663]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:34:56 rum openais[2663]: [CLM ] Members Left: >Aug 17 23:34:56 rum openais[2663]: [CLM ] Members Joined: >Aug 17 23:34:56 rum openais[2663]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:34:56 rum openais[2663]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:34:56 rum openais[2663]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:34:56 rum openais[2663]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:34:56 rum ccsd[2655]: Initial status:: Inquorate >Aug 17 23:34:57 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:34:57 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:34:58 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:34:58 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:34:59 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:34:59 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:00 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:00 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:01 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:01 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:02 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:02 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:03 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:03 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:04 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:04 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:05 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:05 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:06 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:06 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:07 rum dlm_controld[2687]: connect to ccs error -111, check ccsd or cluster status >Aug 17 23:35:07 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:07 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:08 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:08 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:09 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:09 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:10 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:10 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:11 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:11 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:12 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:12 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:13 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:13 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:14 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:14 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:15 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:15 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:16 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:16 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:17 rum dlm_controld[2687]: connect to ccs error -111, check ccsd or cluster status >Aug 17 23:35:17 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:17 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:18 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:18 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:19 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:19 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:20 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:20 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:21 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:21 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:22 rum ccsd[2655]: Cluster is not quorate. Refusing connection. >Aug 17 23:35:22 rum ccsd[2655]: Error while processing connect: Connection refused >Aug 17 23:35:22 rum openais[2663]: [TOTEM] entering GATHER state from 9. >Aug 17 23:35:22 rum openais[2663]: [TOTEM] entering GATHER state from 11. >Aug 17 23:35:22 rum openais[2663]: [TOTEM] Saving state aru b high seq received b >Aug 17 23:35:22 rum openais[2663]: [TOTEM] Storing new sequence id for ring 4db4 >Aug 17 23:35:22 rum openais[2663]: [TOTEM] entering COMMIT state. >Aug 17 23:35:22 rum openais[2663]: [TOTEM] entering RECOVERY state. >Aug 17 23:35:22 rum openais[2663]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:35:22 rum openais[2663]: [TOTEM] previous ring seq 19888 rep 16.181.160.138 >Aug 17 23:35:22 rum openais[2663]: [TOTEM] aru 2b high delivered 2b received flag 1 >Aug 17 23:35:22 rum openais[2663]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:35:22 rum openais[2663]: [TOTEM] previous ring seq 19888 rep 16.181.160.138 >Aug 17 23:35:22 rum openais[2663]: [TOTEM] aru 2b high delivered 2b received flag 1 >Aug 17 23:35:22 rum openais[2663]: [TOTEM] position [2] member 16.181.160.141: >Aug 17 23:35:22 rum openais[2663]: [TOTEM] previous ring seq 19888 rep 16.181.160.141 >Aug 17 23:35:22 rum openais[2663]: [TOTEM] aru b high delivered b received flag 1 >Aug 17 23:35:22 rum openais[2663]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:35:22 rum openais[2663]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:35:22 rum openais[2663]: [CLM ] New Configuration: >Aug 17 23:35:22 rum openais[2663]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:35:22 rum openais[2663]: [CLM ] Members Left: >Aug 17 23:35:22 rum openais[2663]: [CLM ] Members Joined: >Aug 17 23:35:22 rum openais[2663]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:35:22 rum openais[2663]: [CLM ] New Configuration: >Aug 17 23:35:22 rum openais[2663]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:35:22 rum openais[2663]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:35:22 rum openais[2663]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:35:22 rum openais[2663]: [CLM ] Members Left: >Aug 17 23:35:22 rum openais[2663]: [CLM ] Members Joined: >Aug 17 23:35:22 rum openais[2663]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:35:22 rum openais[2663]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:35:22 rum openais[2663]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:35:22 rum openais[2663]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:35:22 rum openais[2663]: [CMAN ] quorum regained, resuming activity >Aug 17 23:35:22 rum openais[2663]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:35:22 rum openais[2663]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:35:22 rum openais[2663]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:35:22 rum openais[2663]: [CPG ] got joinlist message from node 2 >Aug 17 23:35:22 rum openais[2663]: [CPG ] got joinlist message from node 1 >Aug 17 23:35:22 rum openais[2663]: [CPG ] got joinlist message from node 3 >Aug 17 23:35:26 rum kernel: dlm: Using TCP for communications >Aug 17 23:35:26 rum kernel: dlm: got connection from 2 >Aug 17 23:35:26 rum kernel: dlm: got connection from 3 >Aug 17 23:35:28 rum clvmd: Cluster LVM daemon started - connected to CMAN >Aug 17 23:35:28 rum hcid[2792]: Bluetooth HCI daemon >Aug 17 23:35:28 rum kernel: Bluetooth: Core ver 2.10 >Aug 17 23:35:28 rum kernel: NET: Registered protocol family 31 >Aug 17 23:35:28 rum kernel: Bluetooth: HCI device and connection manager initialized >Aug 17 23:35:28 rum kernel: Bluetooth: HCI socket layer initialized >Aug 17 23:35:28 rum hcid[2792]: Register path:/org/bluez fallback:1 >Aug 17 23:35:28 rum kernel: Bluetooth: L2CAP ver 2.8 >Aug 17 23:35:28 rum kernel: Bluetooth: L2CAP socket layer initialized >Aug 17 23:35:28 rum sdpd[2798]: Bluetooth SDP daemon >Aug 17 23:35:28 rum kernel: Bluetooth: RFCOMM socket layer initialized >Aug 17 23:35:28 rum kernel: Bluetooth: RFCOMM TTY layer initialized >Aug 17 23:35:28 rum kernel: Bluetooth: RFCOMM ver 1.8 >Aug 17 23:35:29 rum pcscd: pcscdaemon.c:464:main() pcsc-lite 1.3.1 daemon ready. >Aug 17 23:35:29 rum gfs_controld[2693]: retrieve_plocks: ckpt open error 12 gfs1 >Aug 17 23:35:29 rum kernel: GFS 0.1.19-7.el5 (built Oct 10 2007 20:08:17) installed >Aug 17 23:35:29 rum kernel: Trying to join cluster "lock_dlm", "sgrhel5:gfs1" >Aug 17 23:35:29 rum kernel: Joined cluster. Now mounting FS... >Aug 17 23:35:29 rum kernel: GFS: fsid=sgrhel5:gfs1.0: jid=0: Trying to acquire journal lock... >Aug 17 23:35:29 rum kernel: GFS: fsid=sgrhel5:gfs1.0: jid=0: Looking at journal... >Aug 17 23:35:29 rum kernel: GFS: fsid=sgrhel5:gfs1.0: jid=0: Done >Aug 17 23:35:29 rum kernel: GFS: fsid=sgrhel5:gfs1.0: Scanning for log elements... >Aug 17 23:35:29 rum kernel: GFS: fsid=sgrhel5:gfs1.0: Found 1 unlinked inodes >Aug 17 23:35:29 rum kernel: GFS: fsid=sgrhel5:gfs1.0: Found quota changes for 0 IDs >Aug 17 23:35:29 rum kernel: GFS: fsid=sgrhel5:gfs1.0: Done >Aug 17 23:35:29 rum kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1 >Aug 17 23:35:29 rum hidd[2936]: Bluetooth HID daemon >Aug 17 23:35:29 rum automount[2959]: lookup_read_master: lookup(nisplus): couldn't locat nis+ table auto.master >Aug 17 23:35:30 rum hpiod: 1.6.7 accepting connections at 2208... >Aug 17 23:35:30 rum ntpd[3059]: ntpd 4.2.2p1@1.1570-o Mon Jun 4 15:13:12 UTC 2007 (1) >Aug 17 23:35:30 rum ntpd[3060]: precision = 8.000 usec >Aug 17 23:35:30 rum ntpd[3060]: Listening on interface wildcard, 0.0.0.0#123 Disabled >Aug 17 23:35:30 rum ntpd[3060]: Listening on interface wildcard, ::#123 Disabled >Aug 17 23:35:30 rum ntpd[3060]: Listening on interface eth3, fe80::214:c2ff:fe58:6a8#123 Enabled >Aug 17 23:35:30 rum ntpd[3060]: Listening on interface lo, ::1#123 Enabled >Aug 17 23:35:30 rum ntpd[3060]: Listening on interface eth1, fe80::211:aff:fe5b:6c17#123 Enabled >Aug 17 23:35:30 rum ntpd[3060]: Listening on interface lo, 127.0.0.1#123 Enabled >Aug 17 23:35:30 rum ntpd[3060]: Listening on interface eth3, 16.181.160.141#123 Enabled >Aug 17 23:35:30 rum ntpd[3060]: Listening on interface eth1, 10.10.1.20#123 Enabled >Aug 17 23:35:30 rum ntpd[3060]: kernel time sync status 0040 >Aug 17 23:35:31 rum xinetd[3046]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in. >Aug 17 23:35:31 rum xinetd[3046]: Started working: 5 available services >Aug 17 23:35:31 rum gpm[3101]: *** info [startup.c(95)]: >Aug 17 23:35:31 rum gpm[3101]: Started gpm successfully. Entered daemon mode. >Aug 17 23:35:32 rum kernel: Bridge firewalling registered >Aug 17 23:35:32 rum kernel: ip_tables: (C) 2000-2006 Netfilter Core Team >Aug 17 23:35:33 rum avahi-daemon[3298]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70). >Aug 17 23:35:33 rum avahi-daemon[3298]: Successfully dropped root privileges. >Aug 17 23:35:33 rum avahi-daemon[3298]: avahi-daemon 0.6.16 starting up. >Aug 17 23:35:33 rum avahi-daemon[3298]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns! >Aug 17 23:35:33 rum avahi-daemon[3298]: Successfully called chroot(). >Aug 17 23:35:33 rum avahi-daemon[3298]: Successfully dropped remaining capabilities. >Aug 17 23:35:33 rum kernel: Netfilter messages via NETLINK v0.30. >Aug 17 23:35:33 rum avahi-daemon[3298]: Loading service file /services/sftp-ssh.service. >Aug 17 23:35:33 rum avahi-daemon[3298]: New relevant interface virbr0.IPv6 for mDNS. >Aug 17 23:35:33 rum avahi-daemon[3298]: Joining mDNS multicast group on interface virbr0.IPv6 with address fe80::200:ff:fe00:0. >Aug 17 23:35:33 rum avahi-daemon[3298]: New relevant interface virbr0.IPv4 for mDNS. >Aug 17 23:35:33 rum avahi-daemon[3298]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1. >Aug 17 23:35:33 rum avahi-daemon[3298]: New relevant interface eth1.IPv6 for mDNS. >Aug 17 23:35:33 rum avahi-daemon[3298]: Joining mDNS multicast group on interface eth1.IPv6 with address fe80::211:aff:fe5b:6c17. >Aug 17 23:35:33 rum avahi-daemon[3298]: New relevant interface eth1.IPv4 for mDNS. >Aug 17 23:35:33 rum avahi-daemon[3298]: Joining mDNS multicast group on interface eth1.IPv4 with address 10.10.1.20. >Aug 17 23:35:33 rum avahi-daemon[3298]: New relevant interface eth3.IPv6 for mDNS. >Aug 17 23:35:33 rum avahi-daemon[3298]: Joining mDNS multicast group on interface eth3.IPv6 with address fe80::214:c2ff:fe58:6a8. >Aug 17 23:35:33 rum avahi-daemon[3298]: New relevant interface eth3.IPv4 for mDNS. >Aug 17 23:35:33 rum avahi-daemon[3298]: Joining mDNS multicast group on interface eth3.IPv4 with address 16.181.160.141. >Aug 17 23:35:33 rum ntpd[3060]: frequency initialized 0.000 PPM from /var/lib/ntp/drift >Aug 17 23:35:33 rum kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack >Aug 17 23:35:34 rum avahi-daemon[3298]: Network interface enumeration completed. >Aug 17 23:35:34 rum avahi-daemon[3298]: Registering new address record for fe80::200:ff:fe00:0 on virbr0. >Aug 17 23:35:34 rum avahi-daemon[3298]: Registering new address record for 192.168.122.1 on virbr0. >Aug 17 23:35:34 rum avahi-daemon[3298]: Registering new address record for fe80::211:aff:fe5b:6c17 on eth1. >Aug 17 23:35:34 rum avahi-daemon[3298]: Registering new address record for 10.10.1.20 on eth1. >Aug 17 23:35:34 rum avahi-daemon[3298]: Registering new address record for fe80::214:c2ff:fe58:6a8 on eth3. >Aug 17 23:35:34 rum avahi-daemon[3298]: Registering new address record for 16.181.160.141 on eth3. >Aug 17 23:35:34 rum avahi-daemon[3298]: Registering HINFO record with values 'X86_64'/'LINUX'. >Aug 17 23:35:35 rum avahi-daemon[3298]: Server startup complete. Host name is rum.local. Local service cookie is 1659661967. >Aug 17 23:39:10 rum syslogd 1.4.1: restart. >Aug 17 23:39:10 rum kernel: klogd 1.4.1, log source = /proc/kmsg started. >Aug 17 23:39:10 rum kernel: Linux version 2.6.18-53.el5 (brewbuilder@hs20-bc1-7.build.redhat.com) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Wed Oct 10 16:34:19 EDT 2007 >Aug 17 23:39:10 rum kernel: Command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet > > >-------------- /var/log/messages (vodka) ---------------- > >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] entering GATHER state from 11. >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] Saving state aru 23 high seq received 23 >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] Storing new sequence id for ring 4dac >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] entering COMMIT state. >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] entering RECOVERY state. >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] previous ring seq 19880 rep 16.181.160.138 >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] aru b high delivered b received flag 1 >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] previous ring seq 19880 rep 16.181.160.140 >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] position [2] member 16.181.160.141: >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] previous ring seq 19880 rep 16.181.160.140 >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:27:57 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:27:57 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:27:57 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:27:57 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:27:57 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:27:57 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:27:57 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:27:57 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:27:57 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:27:57 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:27:57 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:27:57 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:27:57 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:27:57 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:27:57 vodka openais[2704]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:27:57 vodka openais[2704]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:27:57 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:27:57 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:27:57 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:27:57 vodka openais[2704]: [CPG ] got joinlist message from node 2 >Aug 17 23:27:57 vodka openais[2704]: [CPG ] got joinlist message from node 1 >Aug 17 23:27:57 vodka openais[2704]: [CPG ] got joinlist message from node 3 >Aug 17 23:28:02 vodka kernel: dlm: connecting to 3 >Aug 17 23:31:48 vodka openais[2704]: [TOTEM] entering GATHER state from 12. >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] entering GATHER state from 0. >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] Saving state aru 48 high seq received 48 >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] Storing new sequence id for ring 4db0 >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] entering COMMIT state. >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] entering RECOVERY state. >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] previous ring seq 19884 rep 16.181.160.138 >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] aru 48 high delivered 48 received flag 1 >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] previous ring seq 19884 rep 16.181.160.138 >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] aru 48 high delivered 48 received flag 1 >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:31:52 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:31:52 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:31:52 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:31:52 vodka kernel: dlm: closing connection to node 1 >Aug 17 23:31:52 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:31:52 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:31:52 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:31:52 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:31:52 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:31:52 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:31:52 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:31:52 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:31:52 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:31:52 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:31:52 vodka openais[2704]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:31:52 vodka openais[2704]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:31:52 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:31:52 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:31:52 vodka openais[2704]: [CPG ] got joinlist message from node 3 >Aug 17 23:31:52 vodka openais[2704]: [CPG ] got joinlist message from node 2 >Aug 17 23:31:52 vodka fenced[2723]: rum.ind.hp.com not a cluster member after 0 sec post_fail_delay >Aug 17 23:31:52 vodka fenced[2723]: fencing node "rum.ind.hp.com" >Aug 17 23:32:57 vodka fenced[2723]: fence "rum.ind.hp.com" success >Aug 17 23:33:02 vodka ccsd[2697]: Attempt to close an unopened CCS descriptor (14880). >Aug 17 23:33:02 vodka ccsd[2697]: Error while processing disconnect: Invalid request descriptor >Aug 17 23:33:02 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Trying to acquire journal lock... >Aug 17 23:33:02 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Looking at journal... >Aug 17 23:33:02 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Acquiring the transaction lock... >Aug 17 23:33:02 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Replaying journal... >Aug 17 23:33:02 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Replayed 0 of 2 blocks >Aug 17 23:33:02 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: replays = 0, skips = 0, sames = 2 >Aug 17 23:33:02 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Journal replayed in 1s >Aug 17 23:33:02 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Done >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] entering GATHER state from 11. >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] Saving state aru 2b high seq received 2b >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] Storing new sequence id for ring 4db4 >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] entering COMMIT state. >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] entering RECOVERY state. >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] previous ring seq 19888 rep 16.181.160.138 >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] aru 2b high delivered 2b received flag 1 >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] previous ring seq 19888 rep 16.181.160.138 >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] aru 2b high delivered 2b received flag 1 >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] position [2] member 16.181.160.141: >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] previous ring seq 19888 rep 16.181.160.141 >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] aru b high delivered b received flag 1 >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:35:24 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:35:24 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:35:24 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:35:24 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:35:24 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:35:24 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:35:24 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:35:24 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:35:24 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:35:24 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:35:24 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:35:24 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:35:24 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:35:24 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:35:24 vodka openais[2704]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:35:24 vodka openais[2704]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:35:24 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:35:24 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:35:24 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:35:24 vodka openais[2704]: [CPG ] got joinlist message from node 2 >Aug 17 23:35:24 vodka openais[2704]: [CPG ] got joinlist message from node 1 >Aug 17 23:35:24 vodka openais[2704]: [CPG ] got joinlist message from node 3 >Aug 17 23:35:29 vodka kernel: dlm: connecting to 1 >Aug 17 23:36:05 vodka openais[2704]: [TOTEM] entering GATHER state from 12. >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] entering GATHER state from 0. >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] Saving state aru 45 high seq received 45 >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] Storing new sequence id for ring 4db8 >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] entering COMMIT state. >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] entering RECOVERY state. >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] previous ring seq 19892 rep 16.181.160.138 >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] aru 45 high delivered 45 received flag 1 >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] previous ring seq 19892 rep 16.181.160.138 >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] aru 45 high delivered 45 received flag 1 >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:36:10 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:36:10 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:36:10 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:36:10 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:36:10 vodka kernel: dlm: closing connection to node 1 >Aug 17 23:36:10 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:36:10 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:36:10 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:36:10 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:36:10 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:36:10 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:36:10 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:36:10 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:36:10 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:36:10 vodka openais[2704]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:36:10 vodka openais[2704]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:36:10 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:36:10 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:36:10 vodka openais[2704]: [CPG ] got joinlist message from node 3 >Aug 17 23:36:10 vodka openais[2704]: [CPG ] got joinlist message from node 2 >Aug 17 23:36:10 vodka fenced[2723]: rum.ind.hp.com not a cluster member after 0 sec post_fail_delay >Aug 17 23:36:10 vodka fenced[2723]: fencing node "rum.ind.hp.com" >Aug 17 23:37:15 vodka fenced[2723]: fence "rum.ind.hp.com" success >Aug 17 23:37:19 vodka ccsd[2697]: Attempt to close an unopened CCS descriptor (16500). >Aug 17 23:37:20 vodka ccsd[2697]: Error while processing disconnect: Invalid request descriptor >Aug 17 23:37:20 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Trying to acquire journal lock... >Aug 17 23:37:20 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Looking at journal... >Aug 17 23:37:20 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Acquiring the transaction lock... >Aug 17 23:37:20 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Replaying journal... >Aug 17 23:37:20 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Replayed 0 of 0 blocks >Aug 17 23:37:20 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: replays = 0, skips = 0, sames = 0 >Aug 17 23:37:20 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Journal replayed in 1s >Aug 17 23:37:20 vodka kernel: GFS: fsid=sgrhel5:gfs1.1: jid=0: Done >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] entering GATHER state from 9. >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] Saving state aru 23 high seq received 23 >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] Storing new sequence id for ring 4dbc >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] entering COMMIT state. >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] entering RECOVERY state. >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] previous ring seq 19896 rep 16.181.160.138 >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] previous ring seq 19896 rep 16.181.160.138 >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] position [2] member 16.181.160.141: >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] previous ring seq 19896 rep 16.181.160.141 >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] aru b high delivered b received flag 1 >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:39:42 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:39:42 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:39:42 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:39:42 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:39:42 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:39:42 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:39:42 vodka openais[2704]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:39:42 vodka openais[2704]: [CLM ] New Configuration: >Aug 17 23:39:42 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:39:42 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:39:42 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:39:42 vodka openais[2704]: [CLM ] Members Left: >Aug 17 23:39:42 vodka openais[2704]: [CLM ] Members Joined: >Aug 17 23:39:42 vodka openais[2704]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:39:42 vodka openais[2704]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:39:42 vodka openais[2704]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:39:42 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:39:42 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:39:42 vodka openais[2704]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:39:42 vodka openais[2704]: [CPG ] got joinlist message from node 2 >Aug 17 23:39:42 vodka openais[2704]: [CPG ] got joinlist message from node 1 >Aug 17 23:39:42 vodka openais[2704]: [CPG ] got joinlist message from node 3 >Aug 17 23:39:47 vodka kernel: dlm: connecting to 1 > >-------------- /var/log/messages (beer) ----------------- > >Aug 17 23:27:57 beer ccsd[2714]: Error while processing connect: Connection refused >Aug 17 23:27:58 beer ccsd[2714]: Cluster is not quorate. Refusing connection. >Aug 17 23:27:58 beer ccsd[2714]: Error while processing connect: Connection refused >Aug 17 23:27:59 beer ccsd[2714]: Cluster is not quorate. Refusing connection. >Aug 17 23:27:59 beer ccsd[2714]: Error while processing connect: Connection refused >Aug 17 23:28:00 beer ccsd[2714]: Cluster is not quorate. Refusing connection. >Aug 17 23:28:00 beer ccsd[2714]: Error while processing connect: Connection refused >Aug 17 23:28:01 beer openais[2721]: [TOTEM] entering GATHER state from 9. >Aug 17 23:28:01 beer openais[2721]: [TOTEM] entering GATHER state from 11. >Aug 17 23:28:01 beer openais[2721]: [TOTEM] Creating commit token because I am the rep. >Aug 17 23:28:01 beer openais[2721]: [TOTEM] Saving state aru b high seq received b >Aug 17 23:28:01 beer openais[2721]: [TOTEM] Storing new sequence id for ring 4dac >Aug 17 23:28:01 beer openais[2721]: [TOTEM] entering COMMIT state. >Aug 17 23:28:01 beer openais[2721]: [TOTEM] entering RECOVERY state. >Aug 17 23:28:01 beer openais[2721]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:28:01 beer openais[2721]: [TOTEM] previous ring seq 19880 rep 16.181.160.138 >Aug 17 23:28:01 beer openais[2721]: [TOTEM] aru b high delivered b received flag 1 >Aug 17 23:28:01 beer openais[2721]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:28:01 beer openais[2721]: [TOTEM] previous ring seq 19880 rep 16.181.160.140 >Aug 17 23:28:01 beer openais[2721]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:28:01 beer openais[2721]: [TOTEM] position [2] member 16.181.160.141: >Aug 17 23:28:01 beer openais[2721]: [TOTEM] previous ring seq 19880 rep 16.181.160.140 >Aug 17 23:28:01 beer openais[2721]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:28:01 beer openais[2721]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:28:01 beer openais[2721]: [TOTEM] Sending initial ORF token >Aug 17 23:28:01 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:28:01 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:28:01 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:28:01 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:28:01 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:28:01 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:28:01 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:28:01 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:28:01 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:28:01 beer openais[2721]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:28:01 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:28:01 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:28:01 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:28:01 beer openais[2721]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:28:01 beer openais[2721]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:28:01 beer openais[2721]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:28:01 beer openais[2721]: [CMAN ] quorum regained, resuming activity >Aug 17 23:28:01 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:28:01 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:28:01 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:28:01 beer openais[2721]: [CPG ] got joinlist message from node 2 >Aug 17 23:28:01 beer openais[2721]: [CPG ] got joinlist message from node 1 >Aug 17 23:28:01 beer openais[2721]: [CPG ] got joinlist message from node 3 >Aug 17 23:28:06 beer kernel: dlm: Using TCP for communications >Aug 17 23:28:06 beer kernel: dlm: got connection from 2 >Aug 17 23:28:06 beer kernel: dlm: connecting to 1 >Aug 17 23:28:06 beer kernel: dlm: got connection from 1 >Aug 17 23:28:07 beer clvmd: Cluster LVM daemon started - connected to CMAN >Aug 17 23:28:07 beer hcid[2851]: Bluetooth HCI daemon >Aug 17 23:28:07 beer kernel: Bluetooth: Core ver 2.10 >Aug 17 23:28:07 beer kernel: NET: Registered protocol family 31 >Aug 17 23:28:07 beer kernel: Bluetooth: HCI device and connection manager initialized >Aug 17 23:28:07 beer kernel: Bluetooth: HCI socket layer initialized >Aug 17 23:28:07 beer hcid[2851]: Register path:/org/bluez fallback:1 >Aug 17 23:28:08 beer kernel: Bluetooth: L2CAP ver 2.8 >Aug 17 23:28:08 beer kernel: Bluetooth: L2CAP socket layer initialized >Aug 17 23:28:08 beer sdpd[2857]: Bluetooth SDP daemon >Aug 17 23:28:08 beer kernel: Bluetooth: RFCOMM socket layer initialized >Aug 17 23:28:08 beer kernel: Bluetooth: RFCOMM TTY layer initialized >Aug 17 23:28:08 beer kernel: Bluetooth: RFCOMM ver 1.8 >Aug 17 23:28:08 beer pcscd: pcscdaemon.c:464:main() pcsc-lite 1.3.1 daemon ready. >Aug 17 23:28:08 beer kernel: GFS 0.1.19-7.el5 (built Oct 10 2007 20:08:17) installed >Aug 17 23:28:08 beer kernel: Trying to join cluster "lock_dlm", "sgrhel5:gfs1" >Aug 17 23:28:08 beer kernel: Joined cluster. Now mounting FS... >Aug 17 23:28:08 beer kernel: GFS: fsid=sgrhel5:gfs1.2: jid=2: Trying to acquire journal lock... >Aug 17 23:28:08 beer kernel: GFS: fsid=sgrhel5:gfs1.2: jid=2: Looking at journal... >Aug 17 23:28:08 beer kernel: GFS: fsid=sgrhel5:gfs1.2: jid=2: Done >Aug 17 23:28:08 beer kernel: GFS: fsid=sgrhel5:gfs1.2: Scanning for log elements... >Aug 17 23:28:08 beer kernel: GFS: fsid=sgrhel5:gfs1.2: Found 1 unlinked inodes >Aug 17 23:28:08 beer kernel: GFS: fsid=sgrhel5:gfs1.2: Found quota changes for 0 IDs >Aug 17 23:28:08 beer kernel: GFS: fsid=sgrhel5:gfs1.2: Done >Aug 17 23:28:08 beer kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1 >Aug 17 23:28:08 beer hidd[2996]: Bluetooth HID daemon >Aug 17 23:28:08 beer automount[3018]: lookup_read_master: lookup(nisplus): couldn't locat nis+ table auto.master >Aug 17 23:28:09 beer hpiod: 1.6.7 accepting connections at 2208... >Aug 17 23:28:09 beer ntpd[3118]: ntpd 4.2.2p1@1.1570-o Mon Jun 4 15:13:12 UTC 2007 (1) >Aug 17 23:28:09 beer ntpd[3119]: precision = 1.000 usec >Aug 17 23:28:09 beer ntpd[3119]: Listening on interface wildcard, 0.0.0.0#123 Disabled >Aug 17 23:28:09 beer ntpd[3119]: Listening on interface wildcard, ::#123 Disabled >Aug 17 23:28:09 beer ntpd[3119]: Listening on interface eth1, fe80::211:aff:fe5b:7077#123 Enabled >Aug 17 23:28:09 beer ntpd[3119]: Listening on interface lo, ::1#123 Enabled >Aug 17 23:28:09 beer ntpd[3119]: Listening on interface eth2, fe80::214:c2ff:fe58:c9f#123 Enabled >Aug 17 23:28:09 beer ntpd[3119]: Listening on interface lo, 127.0.0.1#123 Enabled >Aug 17 23:28:09 beer ntpd[3119]: Listening on interface eth2, 16.181.160.138#123 Enabled >Aug 17 23:28:09 beer ntpd[3119]: Listening on interface eth1, 10.10.1.10#123 Enabled >Aug 17 23:28:09 beer ntpd[3119]: kernel time sync status 0040 >Aug 17 23:28:10 beer xinetd[3105]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in. >Aug 17 23:28:10 beer xinetd[3105]: Started working: 4 available services >Aug 17 23:28:10 beer gpm[3160]: *** info [startup.c(95)]: >Aug 17 23:28:10 beer gpm[3160]: Started gpm successfully. Entered daemon mode. >Aug 17 23:28:11 beer kernel: Bridge firewalling registered >Aug 17 23:28:11 beer kernel: ip_tables: (C) 2000-2006 Netfilter Core Team >Aug 17 23:28:12 beer avahi-daemon[3342]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70). >Aug 17 23:28:12 beer avahi-daemon[3342]: Successfully dropped root privileges. >Aug 17 23:28:12 beer avahi-daemon[3342]: avahi-daemon 0.6.16 starting up. >Aug 17 23:28:12 beer avahi-daemon[3342]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns! >Aug 17 23:28:12 beer avahi-daemon[3342]: Successfully called chroot(). >Aug 17 23:28:12 beer avahi-daemon[3342]: Successfully dropped remaining capabilities. >Aug 17 23:28:12 beer avahi-daemon[3342]: Loading service file /services/sftp-ssh.service. >Aug 17 23:28:12 beer avahi-daemon[3342]: New relevant interface virbr0.IPv6 for mDNS. >Aug 17 23:28:12 beer avahi-daemon[3342]: Joining mDNS multicast group on interface virbr0.IPv6 with address fe80::200:ff:fe00:0. >Aug 17 23:28:12 beer avahi-daemon[3342]: New relevant interface virbr0.IPv4 for mDNS. >Aug 17 23:28:12 beer avahi-daemon[3342]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1. >Aug 17 23:28:12 beer avahi-daemon[3342]: New relevant interface eth1.IPv6 for mDNS. >Aug 17 23:28:12 beer avahi-daemon[3342]: Joining mDNS multicast group on interface eth1.IPv6 with address fe80::211:aff:fe5b:7077. >Aug 17 23:28:12 beer avahi-daemon[3342]: New relevant interface eth1.IPv4 for mDNS. >Aug 17 23:28:12 beer avahi-daemon[3342]: Joining mDNS multicast group on interface eth1.IPv4 with address 10.10.1.10. >Aug 17 23:28:12 beer avahi-daemon[3342]: New relevant interface eth2.IPv6 for mDNS. >Aug 17 23:28:12 beer avahi-daemon[3342]: Joining mDNS multicast group on interface eth2.IPv6 with address fe80::214:c2ff:fe58:c9f. >Aug 17 23:28:12 beer avahi-daemon[3342]: New relevant interface eth2.IPv4 for mDNS. >Aug 17 23:28:12 beer avahi-daemon[3342]: Joining mDNS multicast group on interface eth2.IPv4 with address 16.181.160.138. >Aug 17 23:28:12 beer avahi-daemon[3342]: Network interface enumeration completed. >Aug 17 23:28:12 beer avahi-daemon[3342]: Registering new address record for fe80::200:ff:fe00:0 on virbr0. >Aug 17 23:28:12 beer avahi-daemon[3342]: Registering new address record for 192.168.122.1 on virbr0. >Aug 17 23:28:12 beer avahi-daemon[3342]: Registering new address record for fe80::211:aff:fe5b:7077 on eth1. >Aug 17 23:28:12 beer avahi-daemon[3342]: Registering new address record for 10.10.1.10 on eth1. >Aug 17 23:28:12 beer avahi-daemon[3342]: Registering new address record for fe80::214:c2ff:fe58:c9f on eth2. >Aug 17 23:28:12 beer avahi-daemon[3342]: Registering new address record for 16.181.160.138 on eth2. >Aug 17 23:28:12 beer avahi-daemon[3342]: Registering HINFO record with values 'X86_64'/'LINUX'. >Aug 17 23:28:13 beer kernel: Netfilter messages via NETLINK v0.30. >Aug 17 23:28:13 beer kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack >Aug 17 23:28:13 beer avahi-daemon[3342]: Server startup complete. Host name is beer.local. Local service cookie is 1922612436. >Aug 17 23:28:14 beer avahi-daemon[3342]: Service "SFTP File Transfer on beer" (/services/sftp-ssh.service) successfully established. >Aug 17 23:28:15 beer dnsmasq[3415]: started, version 2.39 cachesize 150 >Aug 17 23:28:15 beer dnsmasq[3415]: compile time options: IPv6 GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP >Aug 17 23:28:15 beer dnsmasq[3415]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h >Aug 17 23:28:15 beer dnsmasq[3415]: reading /etc/resolv.conf >Aug 17 23:28:15 beer dnsmasq[3415]: using nameserver 16.110.135.51#53 >Aug 17 23:28:15 beer dnsmasq[3415]: bad address at /etc/hosts line 13 >Aug 17 23:28:15 beer dnsmasq[3415]: read /etc/hosts - 7 addresses >Aug 17 23:28:17 beer ntpd[3119]: frequency initialized 0.000 PPM from /var/lib/ntp/drift >Aug 17 23:28:18 beer cmcluster: AUTOSTART_CMCLD not set to 1 in /usr/local/cmcluster/conf/cmcluster.rc, exiting >Aug 17 23:28:18 beer modclusterd: startup succeeded >Aug 17 23:28:18 beer oddjobd: oddjobd startup succeeded >Aug 17 23:28:18 beer saslauthd[3676]: detach_tty : master pid is: 3676 >Aug 17 23:28:18 beer saslauthd[3676]: ipc_init : listening on socket: /var/run/saslauthd/mux >Aug 17 23:28:18 beer ricci: startup succeeded >Aug 17 23:28:19 beer smartd[3706]: smartd version 5.36 [x86_64-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen >Aug 17 23:28:19 beer smartd[3706]: Home page is http://smartmontools.sourceforge.net/ >Aug 17 23:28:19 beer smartd[3706]: Opened configuration file /etc/smartd.conf >Aug 17 23:28:19 beer smartd[3706]: Configuration file /etc/smartd.conf parsed. >Aug 17 23:28:19 beer smartd[3706]: Device: /dev/sda, opened >Aug 17 23:28:19 beer smartd[3706]: Device: /dev/sda, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on SMART features >Aug 17 23:28:19 beer smartd[3706]: Unable to register SCSI device /dev/sda at line 32 of file /etc/smartd.conf >Aug 17 23:28:19 beer smartd[3706]: Device /dev/sda not available >Aug 17 23:28:19 beer smartd[3706]: Device: /dev/sdb, opened >Aug 17 23:28:19 beer smartd[3706]: Device: /dev/sdb, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdb' to turn on SMART features >Aug 17 23:28:19 beer smartd[3706]: Unable to register SCSI device /dev/sdb at line 33 of file /etc/smartd.conf >Aug 17 23:28:19 beer smartd[3706]: Device /dev/sdb not available >Aug 17 23:28:19 beer smartd[3706]: Device: /dev/sdc, opened >Aug 17 23:28:19 beer smartd[3706]: Device: /dev/sdc, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdc' to turn on SMART features >Aug 17 23:28:19 beer smartd[3706]: Unable to register SCSI device /dev/sdc at line 34 of file /etc/smartd.conf >Aug 17 23:28:19 beer smartd[3706]: Device /dev/sdc not available >Aug 17 23:28:19 beer smartd[3706]: Monitoring 0 ATA and 0 SCSI devices >Aug 17 23:28:19 beer smartd[3708]: smartd has fork()ed into background mode. New PID=3708. >Aug 17 23:28:19 beer pcscd: winscard.c:219:SCardConnect() Reader E-Gate 0 0 Not Found >Aug 17 23:28:19 beer last message repeated 3 times >Aug 17 23:28:19 beer kernel: mtrr: type mismatch for f6000000,800000 old: uncachable new: write-combining >Aug 17 23:28:19 beer kernel: mtrr: type mismatch for f6000000,800000 old: uncachable new: write-combining >Aug 17 23:31:34 beer ntpd[3119]: synchronized to LOCAL(0), stratum 10 >Aug 17 23:31:34 beer ntpd[3119]: kernel time sync enabled 0001 >Aug 17 23:31:51 beer openais[2721]: [TOTEM] The token was lost in the OPERATIONAL state. >Aug 17 23:31:51 beer openais[2721]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). >Aug 17 23:31:51 beer openais[2721]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). >Aug 17 23:31:51 beer openais[2721]: [TOTEM] entering GATHER state from 2. >Aug 17 23:31:56 beer openais[2721]: [TOTEM] entering GATHER state from 11. >Aug 17 23:31:56 beer openais[2721]: [TOTEM] Creating commit token because I am the rep. >Aug 17 23:31:56 beer openais[2721]: [TOTEM] Saving state aru 48 high seq received 48 >Aug 17 23:31:56 beer openais[2721]: [TOTEM] Storing new sequence id for ring 4db0 >Aug 17 23:31:56 beer openais[2721]: [TOTEM] entering COMMIT state. >Aug 17 23:31:56 beer openais[2721]: [TOTEM] entering RECOVERY state. >Aug 17 23:31:56 beer openais[2721]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:31:56 beer openais[2721]: [TOTEM] previous ring seq 19884 rep 16.181.160.138 >Aug 17 23:31:56 beer openais[2721]: [TOTEM] aru 48 high delivered 48 received flag 1 >Aug 17 23:31:56 beer openais[2721]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:31:56 beer openais[2721]: [TOTEM] previous ring seq 19884 rep 16.181.160.138 >Aug 17 23:31:56 beer openais[2721]: [TOTEM] aru 48 high delivered 48 received flag 1 >Aug 17 23:31:56 beer openais[2721]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:31:56 beer openais[2721]: [TOTEM] Sending initial ORF token >Aug 17 23:31:56 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:31:56 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:31:56 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:31:56 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:31:56 beer kernel: dlm: closing connection to node 1 >Aug 17 23:31:56 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:31:56 beer openais[2721]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:31:56 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:31:56 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:31:56 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:31:56 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:31:56 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:31:56 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:31:56 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:31:56 beer openais[2721]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:31:56 beer openais[2721]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:31:56 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:31:56 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:31:56 beer openais[2721]: [CPG ] got joinlist message from node 3 >Aug 17 23:31:56 beer openais[2721]: [CPG ] got joinlist message from node 2 >Aug 17 23:31:56 beer fenced[2740]: fencing deferred to vodka.ind.hp.com >Aug 17 23:33:05 beer kernel: GFS: fsid=sgrhel5:gfs1.2: jid=0: Trying to acquire journal lock... >Aug 17 23:33:06 beer kernel: GFS: fsid=sgrhel5:gfs1.2: jid=0: Busy >Aug 17 23:35:28 beer openais[2721]: [TOTEM] entering GATHER state from 11. >Aug 17 23:35:28 beer openais[2721]: [TOTEM] Creating commit token because I am the rep. >Aug 17 23:35:28 beer openais[2721]: [TOTEM] Saving state aru 2b high seq received 2b >Aug 17 23:35:28 beer openais[2721]: [TOTEM] Storing new sequence id for ring 4db4 >Aug 17 23:35:28 beer openais[2721]: [TOTEM] entering COMMIT state. >Aug 17 23:35:28 beer openais[2721]: [TOTEM] entering RECOVERY state. >Aug 17 23:35:28 beer openais[2721]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:35:28 beer openais[2721]: [TOTEM] previous ring seq 19888 rep 16.181.160.138 >Aug 17 23:35:28 beer openais[2721]: [TOTEM] aru 2b high delivered 2b received flag 1 >Aug 17 23:35:28 beer openais[2721]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:35:28 beer openais[2721]: [TOTEM] previous ring seq 19888 rep 16.181.160.138 >Aug 17 23:35:28 beer openais[2721]: [TOTEM] aru 2b high delivered 2b received flag 1 >Aug 17 23:35:28 beer openais[2721]: [TOTEM] position [2] member 16.181.160.141: >Aug 17 23:35:28 beer openais[2721]: [TOTEM] previous ring seq 19888 rep 16.181.160.141 >Aug 17 23:35:28 beer openais[2721]: [TOTEM] aru b high delivered b received flag 1 >Aug 17 23:35:28 beer openais[2721]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:35:28 beer openais[2721]: [TOTEM] Sending initial ORF token >Aug 17 23:35:28 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:35:28 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:35:28 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:35:28 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:35:28 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:35:28 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:35:28 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:35:28 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:35:28 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:35:28 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:35:28 beer openais[2721]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:35:28 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:35:28 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:35:28 beer openais[2721]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:35:28 beer openais[2721]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:35:28 beer openais[2721]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:35:28 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:35:28 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:35:28 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:35:28 beer openais[2721]: [CPG ] got joinlist message from node 2 >Aug 17 23:35:28 beer openais[2721]: [CPG ] got joinlist message from node 1 >Aug 17 23:35:28 beer openais[2721]: [CPG ] got joinlist message from node 3 >Aug 17 23:35:32 beer kernel: dlm: connecting to 1 >Aug 17 23:36:09 beer openais[2721]: [TOTEM] The token was lost in the OPERATIONAL state. >Aug 17 23:36:09 beer openais[2721]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). >Aug 17 23:36:09 beer openais[2721]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). >Aug 17 23:36:09 beer openais[2721]: [TOTEM] entering GATHER state from 2. >Aug 17 23:36:13 beer openais[2721]: [TOTEM] entering GATHER state from 11. >Aug 17 23:36:14 beer openais[2721]: [TOTEM] Creating commit token because I am the rep. >Aug 17 23:36:14 beer openais[2721]: [TOTEM] Saving state aru 45 high seq received 45 >Aug 17 23:36:14 beer openais[2721]: [TOTEM] Storing new sequence id for ring 4db8 >Aug 17 23:36:14 beer openais[2721]: [TOTEM] entering COMMIT state. >Aug 17 23:36:14 beer openais[2721]: [TOTEM] entering RECOVERY state. >Aug 17 23:36:14 beer openais[2721]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:36:14 beer openais[2721]: [TOTEM] previous ring seq 19892 rep 16.181.160.138 >Aug 17 23:36:14 beer openais[2721]: [TOTEM] aru 45 high delivered 45 received flag 1 >Aug 17 23:36:14 beer openais[2721]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:36:14 beer openais[2721]: [TOTEM] previous ring seq 19892 rep 16.181.160.138 >Aug 17 23:36:14 beer openais[2721]: [TOTEM] aru 45 high delivered 45 received flag 1 >Aug 17 23:36:14 beer openais[2721]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:36:14 beer openais[2721]: [TOTEM] Sending initial ORF token >Aug 17 23:36:14 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:36:14 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:36:14 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:36:14 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:36:14 beer kernel: dlm: closing connection to node 1 >Aug 17 23:36:14 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:36:14 beer openais[2721]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:36:14 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:36:14 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:36:14 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:36:14 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:36:14 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:36:14 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:36:14 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:36:14 beer openais[2721]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:36:14 beer openais[2721]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:36:14 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:36:14 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:36:14 beer openais[2721]: [CPG ] got joinlist message from node 3 >Aug 17 23:36:14 beer openais[2721]: [CPG ] got joinlist message from node 2 >Aug 17 23:36:14 beer fenced[2740]: fencing deferred to vodka.ind.hp.com >Aug 17 23:37:23 beer kernel: GFS: fsid=sgrhel5:gfs1.2: jid=0: Trying to acquire journal lock... >Aug 17 23:37:23 beer kernel: GFS: fsid=sgrhel5:gfs1.2: jid=0: Busy >Aug 17 23:39:45 beer openais[2721]: [TOTEM] entering GATHER state from 9. >Aug 17 23:39:45 beer openais[2721]: [TOTEM] Creating commit token because I am the rep. >Aug 17 23:39:45 beer openais[2721]: [TOTEM] Saving state aru 23 high seq received 23 >Aug 17 23:39:45 beer openais[2721]: [TOTEM] Storing new sequence id for ring 4dbc >Aug 17 23:39:45 beer openais[2721]: [TOTEM] entering COMMIT state. >Aug 17 23:39:45 beer openais[2721]: [TOTEM] entering RECOVERY state. >Aug 17 23:39:45 beer openais[2721]: [TOTEM] position [0] member 16.181.160.138: >Aug 17 23:39:45 beer openais[2721]: [TOTEM] previous ring seq 19896 rep 16.181.160.138 >Aug 17 23:39:45 beer openais[2721]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:39:45 beer openais[2721]: [TOTEM] position [1] member 16.181.160.140: >Aug 17 23:39:45 beer openais[2721]: [TOTEM] previous ring seq 19896 rep 16.181.160.138 >Aug 17 23:39:45 beer openais[2721]: [TOTEM] aru 23 high delivered 23 received flag 1 >Aug 17 23:39:45 beer openais[2721]: [TOTEM] position [2] member 16.181.160.141: >Aug 17 23:39:45 beer openais[2721]: [TOTEM] previous ring seq 19896 rep 16.181.160.141 >Aug 17 23:39:45 beer openais[2721]: [TOTEM] aru b high delivered b received flag 1 >Aug 17 23:39:45 beer openais[2721]: [TOTEM] Did not need to originate any messages in recovery. >Aug 17 23:39:45 beer openais[2721]: [TOTEM] Sending initial ORF token >Aug 17 23:39:45 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:39:45 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:39:45 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:39:45 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:39:45 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:39:45 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:39:45 beer openais[2721]: [CLM ] CLM CONFIGURATION CHANGE >Aug 17 23:39:45 beer openais[2721]: [CLM ] New Configuration: >Aug 17 23:39:45 beer openais[2721]: [CLM ] r(0) ip(16.181.160.138) >Aug 17 23:39:45 beer openais[2721]: [CLM ] r(0) ip(16.181.160.140) >Aug 17 23:39:45 beer openais[2721]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:39:45 beer openais[2721]: [CLM ] Members Left: >Aug 17 23:39:45 beer openais[2721]: [CLM ] Members Joined: >Aug 17 23:39:45 beer openais[2721]: [CLM ] r(0) ip(16.181.160.141) >Aug 17 23:39:45 beer openais[2721]: [SYNC ] This node is within the primary component and will provide service. >Aug 17 23:39:45 beer openais[2721]: [TOTEM] entering OPERATIONAL state. >Aug 17 23:39:45 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.138 >Aug 17 23:39:45 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.140 >Aug 17 23:39:45 beer openais[2721]: [CLM ] got nodejoin message 16.181.160.141 >Aug 17 23:39:45 beer openais[2721]: [CPG ] got joinlist message from node 2 >Aug 17 23:39:45 beer openais[2721]: [CPG ] got joinlist message from node 1 >Aug 17 23:39:45 beer openais[2721]: [CPG ] got joinlist message from node 3 >Aug 17 23:39:50 beer kernel: dlm: connecting to 1 > > > >Test script (/root/dwn/rhcs_failure.sh) for beer >------------------------------------------------ > ># ># 1. if GFS directory is not mounted ># exit ># 2. Wait in loop until all the nodes are cluster members ># 3. Create a lock directory in the GFS FS ># NOTE: This is used for serializing LAN failure test accross the 3 nodes. ># It ensure that only one node can fail the LAN interface at time ># 4. Sleep for Random time ># 5. Verify that it is still a 3 node cluster else go back to 4. ># 5. remove the lock directory ># 6. If successfully ># Bring down the heartbeat interface ># else ># go to 4 ># > >PATH=$PATH:/usr/local/cmcluster/bin >HB=eth2 >FLAG=1 >GFS=1 >NODE="beer" >NOT_MOUNTED=1 >MOUNT_DIR="/mnt/apache" > >echo -e "\nTest program started - $(date '+%b %e %T') - Node \"$(hostname)\" "; > >echo -e "\nChecking if $MOUNT_DIR is mounted on startup ..."; > >mount |grep ${MOUNT_DIR} >NOT_MOUNTED=$? >if (( $NOT_MOUNTED )) >then > echo "GFS $MOUNT_DIR is NOT mounted on host startup. Exiting ..."; > exit -1 >else > echo "GFS $MOUNT_DIR is mounted on host startup"; >fi >echo -e "\nSleeping for 100 secs..." >sleep 100; ># wait in a loop till a 3 node GFS cluster and current node is SG node > >while [ 1 ] >do > if (( $GFS )) > then > GFS=$( cman_tool status | grep "Total votes" | awk '{print $3}' ); > if [ "$GFS" == "3" ] > then > GFS=0; > echo "3 node RHCS formed -- $(date '+%b %e %T')"; > else > echo "Currently $GFS node cluster"; > echo "Sleeping for other nodes to join -- $(date '+%b %e %T')"; > sleep 10; > GFS=1; > fi > else > echo "Creating lockdir GFS directory"; > mkdir /mnt/apache/lockdir; > break; > fi >done > >while [ 1 ] >do > sleep 4; > # sleep a random amount of time (0-7 secs) to ensure all nodes reboot > RR=$RANDOM > RNDM=`echo $RR/4000 | bc` > echo -e "\nsleeping for Random time: $RNDM secs -- $(date '+%b %e %T') - Node \"$(hostname)\" "; > sleep $RNDM; > sleep 10; > > GFS=$( cman_tool status | grep "Total votes" | awk '{print $3}' ); > if [ $GFS == "3" ] > then > echo -e "\n$GFS node RHCS present"; > echo "WARNING: Other nodes GFS FS might not be mounted yet"; > echo "Removing lockdir: - $(date '+%b %e %T') - Node \"$(hostname)\" "; > rmdir /mnt/apache/lockdir; > > FLAG=$? > if [ $FLAG != "0" ] > then > echo "Failed to remove lockdir : $FLAG -- $(date '+%b %e %T') - Node \"$(hostname)\" "; > echo "One server has lost the heartbeat"; > else > echo "Successfully removed lockdir : $FLAG -- $(date '+%b %e %T') - Node \"$(hostname)\" "; > echo -e "\nThe heart beat link $HB is brought down on node $NODE"; > ifconfig $HB down; > #/sbin/reboot > while [ 1 ] > do > date; > sleep 1; > done > fi > else > echo -e "\n$GFS node RHCS"; > fi >done > >--------------------------------------------- > >Note: > >1. THe exact script is used for the nodes, vodka and rum after changing the HB and NODE flags in the > script. > >2. This script is started via init.d at bootup time on all the machines. > ln -s /etc/init.d/rhcsrc /etc/rc5.d/S99rhcsrc > >3. Log file is /tmp/dwn/rhcs_failure.log on each node > >4. cman, clvmd and gfs are started on bootup time (chkconfig is enabled) > >5. Only one node is allowed to bring down the interface at a time. THe node that successfully > removes a directory in the GFS FS will bring down the interface. The other nodes will wait > till the node rebooted and creates the directory in the GFS FS.
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=314735">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 459731
: 314735