Attachment 312922 Details for Bug 457107 – log from hayes-02

log from hayes-02

hayes-02.log (text/plain), 24.71 KB, created by Corey Marthaler on 2008-07-29 19:07:34 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2008-07-29 19:07:34 UTC

Size: 24.71 KB

patch

obsolete

>Jul 29 13:37:16 hayes-02 kernel: DLM (built Jul 17 2008 18:05:19) installed
>Jul 29 13:37:16 hayes-02 kernel: GFS2 (built Jul 17 2008 18:05:45) installed
>Jul 29 13:37:16 hayes-02 kernel: Lock_DLM (built Jul 17 2008 18:05:51) installed
>Jul 29 13:37:17 hayes-02 ccsd[3127]: Starting ccsd 2.0.86: 
>Jul 29 13:37:17 hayes-02 ccsd[3127]:  Built: Jul 17 2008 15:38:59 
>Jul 29 13:37:17 hayes-02 ccsd[3127]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved. 
>Jul 29 13:37:17 hayes-02 ccsd[3127]: cluster.conf (cluster name = HAYES, version = 2) found. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] AIS Executive Service: started and ready to provide service. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Using default multicast address of 239.192.8.209 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_cpg loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_cfg loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais configuration service' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_msg loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais message service B.01.01' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_lck loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_evt loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais event service B.01.01' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_ckpt loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_amf loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_clm loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_evs loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] openais component openais_cman loaded. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] send threads (0 threads) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] RRP token expired timeout (495 ms) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] RRP token problem counter (2000 ms) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] RRP threshold (10 problem count) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] RRP mode set to none. 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] heartbeat_failures_allowed (0) 
>Jul 29 13:37:19 hayes-02 openais[3135]: [TOTEM] max_network_delay (50 ms) 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] The network interface [10.15.89.136] is now up. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Created or loaded sequence id 660.10.15.89.136 for this ring. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering GATHER state from 15. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais event service B.01.01' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais message service B.01.01' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais configuration service' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CMAN ] CMAN 2.0.86 (built Jul 17 2008 15:39:03) started 
>Jul 29 13:37:20 hayes-02 openais[3135]: [SYNC ] Not using a virtual synchrony filter. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Creating commit token because I am the rep. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Saving state aru 0 high seq received 0 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Storing new sequence id for ring 298 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering COMMIT state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering RECOVERY state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] position [0] member 10.15.89.136: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] previous ring seq 660 rep 10.15.89.136 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] aru 0 high delivered 0 received flag 1 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Did not need to originate any messages in recovery. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Sending initial ORF token 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [SYNC ] This node is within the primary component and will provide service. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering OPERATIONAL state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] got nodejoin message 10.15.89.136 
>Jul 29 13:37:20 hayes-02 ccsd[3127]: Initial status:: Inquorate 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering GATHER state from 11. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Saving state aru a high seq received a 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Storing new sequence id for ring 29c 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering COMMIT state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering RECOVERY state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] position [0] member 10.15.89.135: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] previous ring seq 664 rep 10.15.89.135 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] aru b high delivered b received flag 1 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] position [1] member 10.15.89.136: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] previous ring seq 664 rep 10.15.89.136 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] aru a high delivered a received flag 1 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Did not need to originate any messages in recovery. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [SYNC ] This node is within the primary component and will provide service. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering OPERATIONAL state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CMAN ] quorum regained, resuming activity 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] got nodejoin message 10.15.89.135 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] got nodejoin message 10.15.89.136 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CPG  ] got joinlist message from node 1 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering GATHER state from 11. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Saving state aru 18 high seq received 18 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Storing new sequence id for ring 2a0 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering COMMIT state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering RECOVERY state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] position [0] member 10.15.89.135: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] previous ring seq 668 rep 10.15.89.135 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] aru 18 high delivered 18 received flag 1 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] position [1] member 10.15.89.136: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] previous ring seq 668 rep 10.15.89.135 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] aru 18 high delivered 18 received flag 1 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] position [2] member 10.15.89.137: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] previous ring seq 656 rep 10.15.89.137 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] aru a high delivered a received flag 1 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] Did not need to originate any messages in recovery. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jul 29 13:37:20 hayes-02 openais[3135]: [SYNC ] This node is within the primary component and will provide service. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [TOTEM] entering OPERATIONAL state. 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] got nodejoin message 10.15.89.135 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] got nodejoin message 10.15.89.136 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CLM  ] got nodejoin message 10.15.89.137 
>Jul 29 13:37:20 hayes-02 openais[3135]: [CPG  ] got joinlist message from node 1 
>Jul 29 13:37:31 hayes-02 kernel: dlm: Using TCP for communications
>Jul 29 13:37:31 hayes-02 kernel: dlm: connecting to 3
>Jul 29 13:37:31 hayes-02 kernel: dlm: got connection from 3
>Jul 29 13:37:31 hayes-02 kernel: dlm: connecting to 1
>Jul 29 13:37:31 hayes-02 kernel: dlm: got connection from 1
>Jul 29 13:37:32 hayes-02 clvmd: Cluster LVM daemon started - connected to CMAN
>Jul 29 13:37:54 hayes-02 clogd[3262]: Starting clogd: 
>Jul 29 13:37:54 hayes-02 clogd[3262]:  Built: Jul 24 2008 16:59:01  
>Jul 29 13:37:58 hayes-02 kernel: aoe: aoe_init: AoE v22 initialised.
>Jul 29 13:37:58 hayes-02 kernel: aoe: 1.1 isbusy
>Jul 29 13:37:58 hayes-02 kernel: aoe: 0030488d63d2 e1.1 v4467 has 19046937725 sectors
>Jul 29 13:37:59 hayes-02 kernel:  etherd/e1.1: p1 p2 p3 p4 p5
>Jul 29 13:37:59 hayes-02 kernel: aoe: 1.1 isbusy
>Jul 29 13:37:59 hayes-02 last message repeated 2 times
>Jul 29 13:38:04 hayes-02 kernel: dlm: Using TCP for communications
>Jul 29 13:38:04 hayes-02 kernel: dlm: connecting to 3
>Jul 29 13:38:04 hayes-02 kernel: dlm: connecting to 1
>Jul 29 13:38:04 hayes-02 clvmd: Cluster LVM daemon started - connected to CMAN
>Jul 29 13:38:04 hayes-02 kernel: dlm: got connection from 3
>Jul 29 13:38:04 hayes-02 kernel: dlm: got connection from 1
>Jul 29 13:38:04 hayes-02 kernel: aoe: 1.1 isbusy
>Jul 29 13:38:58 hayes-02 last message repeated 5 times
>Jul 29 13:40:58 hayes-02 last message repeated 2 times
>Jul 29 13:42:58 hayes-02 last message repeated 2 times
>Jul 29 13:43:58 hayes-02 kernel: aoe: 1.1 isbusy
>Jul 29 13:44:12 hayes-02 openais[3135]: [TOTEM] The token was lost in the OPERATIONAL state. 
>Jul 29 13:44:12 hayes-02 openais[3135]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
>Jul 29 13:44:12 hayes-02 openais[3135]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jul 29 13:44:12 hayes-02 openais[3135]: [TOTEM] entering GATHER state from 2. 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] entering GATHER state from 11. 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] Creating commit token because I am the rep. 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] Saving state aru d7 high seq received d7 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] Storing new sequence id for ring 2a4 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] entering COMMIT state. 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] entering RECOVERY state. 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] position [0] member 10.15.89.136: 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] previous ring seq 672 rep 10.15.89.135 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] aru d7 high delivered d7 received flag 1 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] position [1] member 10.15.89.137: 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] previous ring seq 672 rep 10.15.89.135 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] aru d7 high delivered d7 received flag 1 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] Did not need to originate any messages in recovery. 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] Sending initial ORF token 
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jul 29 13:44:17 hayes-02 kernel: dlm: closing connection to node 1
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:44:17 hayes-02 kernel: dlm: closing connection to node 3
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:44:17 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:44:17 hayes-02 openais[3135]: [SYNC ] This node is within the primary component and will provide service. 
>Jul 29 13:44:17 hayes-02 openais[3135]: [TOTEM] entering OPERATIONAL state. 
>Jul 29 13:44:17 hayes-02 openais[3135]: [MAIN ] Killing node hayes-03 because it has rejoined the cluster with existing state 
>Jul 29 13:44:17 hayes-02 openais[3135]: [CMAN ] quorum lost, blocking activity 
>Jul 29 13:44:17 hayes-02 groupd[3148]: cpg_mcast_joined retry 100
>Jul 29 13:44:17 hayes-02 groupd[3148]: cpg_mcast_joined retry 200
>Jul 29 13:44:17 hayes-02 groupd[3148]: cpg_mcast_joined retry 300
>Jul 29 13:44:17 hayes-02 groupd[3148]: cpg_mcast_joined retry 400
>Jul 29 13:44:18 hayes-02 groupd[3148]: cpg_mcast_joined retry 500
>Jul 29 13:44:18 hayes-02 groupd[3148]: cpg_mcast_joined retry 600
>Jul 29 13:44:18 hayes-02 groupd[3148]: cpg_mcast_joined retry 700
>Jul 29 13:44:18 hayes-02 groupd[3148]: cpg_mcast_joined retry 800
>Jul 29 13:44:18 hayes-02 groupd[3148]: cpg_mcast_joined retry 900
>Jul 29 13:44:19 hayes-02 groupd[3148]: cpg_mcast_joined retry 1000
>Jul 29 13:44:19 hayes-02 groupd[3148]: cpg_mcast_joined retry 1100
>Jul 29 13:44:19 hayes-02 groupd[3148]: cpg_mcast_joined retry 1200
>Jul 29 13:44:19 hayes-02 groupd[3148]: cpg_mcast_joined retry 1300
>Jul 29 13:44:19 hayes-02 groupd[3148]: cpg_mcast_joined retry 1400
>Jul 29 13:44:20 hayes-02 groupd[3148]: cpg_mcast_joined retry 1500
>Jul 29 13:44:20 hayes-02 groupd[3148]: cpg_mcast_joined retry 1600
>Jul 29 13:44:20 hayes-02 groupd[3148]: cpg_mcast_joined retry 1700
>Jul 29 13:44:20 hayes-02 groupd[3148]: cpg_mcast_joined retry 1800
>Jul 29 13:44:20 hayes-02 groupd[3148]: cpg_mcast_joined retry 1900
>Jul 29 13:44:21 hayes-02 groupd[3148]: cpg_mcast_joined retry 2000
>Jul 29 13:44:21 hayes-02 groupd[3148]: cpg_mcast_joined retry 2100
>Jul 29 13:44:21 hayes-02 groupd[3148]: cpg_mcast_joined retry 2200
>Jul 29 13:44:21 hayes-02 groupd[3148]: cpg_mcast_joined retry 2300
>Jul 29 13:44:21 hayes-02 groupd[3148]: cpg_mcast_joined retry 2400
>Jul 29 13:44:22 hayes-02 groupd[3148]: cpg_mcast_joined retry 2500
>Jul 29 13:44:22 hayes-02 groupd[3148]: cpg_mcast_joined retry 2600
>Jul 29 13:44:22 hayes-02 groupd[3148]: cpg_mcast_joined retry 2700
>Jul 29 13:44:22 hayes-02 groupd[3148]: cpg_mcast_joined retry 2800
>Jul 29 13:44:22 hayes-02 groupd[3148]: cpg_mcast_joined retry 2900
>Jul 29 13:44:23 hayes-02 groupd[3148]: cpg_mcast_joined retry 3000
>Jul 29 13:44:23 hayes-02 groupd[3148]: cpg_mcast_joined retry 3100
>Jul 29 13:44:23 hayes-02 groupd[3148]: cpg_mcast_joined retry 3200
>Jul 29 13:44:23 hayes-02 groupd[3148]: cpg_mcast_joined retry 3300
>Jul 29 13:44:23 hayes-02 groupd[3148]: cpg_mcast_joined retry 3400
>Jul 29 13:44:24 hayes-02 groupd[3148]: cpg_mcast_joined retry 3500
>Jul 29 13:44:24 hayes-02 groupd[3148]: cpg_mcast_joined retry 3600
>Jul 29 13:44:24 hayes-02 groupd[3148]: cpg_mcast_joined retry 3700
>Jul 29 13:44:24 hayes-02 groupd[3148]: cpg_mcast_joined retry 3800
>Jul 29 13:44:25 hayes-02 groupd[3148]: cpg_mcast_joined retry 3900
>Jul 29 13:44:25 hayes-02 groupd[3148]: cpg_mcast_joined retry 4000
>Jul 29 13:44:25 hayes-02 groupd[3148]: cpg_mcast_joined retry 4100
>Jul 29 13:44:25 hayes-02 groupd[3148]: cpg_mcast_joined retry 4200
>Jul 29 13:44:25 hayes-02 groupd[3148]: cpg_mcast_joined retry 4300
>Jul 29 13:44:26 hayes-02 groupd[3148]: cpg_mcast_joined retry 4400
>Jul 29 13:44:26 hayes-02 groupd[3148]: cpg_mcast_joined retry 4500
>Jul 29 13:44:26 hayes-02 groupd[3148]: cpg_mcast_joined retry 4600
>Jul 29 13:44:26 hayes-02 groupd[3148]: cpg_mcast_joined retry 4700
>Jul 29 13:44:26 hayes-02 groupd[3148]: cpg_mcast_joined retry 4800
>Jul 29 13:44:27 hayes-02 groupd[3148]: cpg_mcast_joined retry 4900
>Jul 29 13:44:27 hayes-02 openais[3135]: [TOTEM] The token was lost in the OPERATIONAL state. 
>Jul 29 13:44:27 hayes-02 openais[3135]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
>Jul 29 13:44:27 hayes-02 openais[3135]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jul 29 13:44:27 hayes-02 openais[3135]: [TOTEM] entering GATHER state from 2. 
>Jul 29 13:44:27 hayes-02 groupd[3148]: cpg_mcast_joined retry 5000
>Jul 29 13:44:27 hayes-02 groupd[3148]: cpg_mcast_joined retry 5100
>Jul 29 13:44:27 hayes-02 groupd[3148]: cpg_mcast_joined retry 5200
>Jul 29 13:44:27 hayes-02 groupd[3148]: cpg_mcast_joined retry 5300
>Jul 29 13:44:28 hayes-02 groupd[3148]: cpg_mcast_joined retry 5400
>Jul 29 13:44:28 hayes-02 groupd[3148]: cpg_mcast_joined retry 5500
>Jul 29 13:44:28 hayes-02 groupd[3148]: cpg_mcast_joined retry 5600
>Jul 29 13:44:28 hayes-02 groupd[3148]: cpg_mcast_joined retry 5700
>Jul 29 13:44:28 hayes-02 groupd[3148]: cpg_mcast_joined retry 5800
>Jul 29 13:44:29 hayes-02 groupd[3148]: cpg_mcast_joined retry 5900
>Jul 29 13:44:29 hayes-02 groupd[3148]: cpg_mcast_joined retry 6000
>Jul 29 13:44:29 hayes-02 groupd[3148]: cpg_mcast_joined retry 6100
>Jul 29 13:44:29 hayes-02 groupd[3148]: cpg_mcast_joined retry 6200
>Jul 29 13:44:29 hayes-02 groupd[3148]: cpg_mcast_joined retry 6300
>Jul 29 13:44:30 hayes-02 groupd[3148]: cpg_mcast_joined retry 6400
>Jul 29 13:44:30 hayes-02 groupd[3148]: cpg_mcast_joined retry 6500
>Jul 29 13:44:30 hayes-02 groupd[3148]: cpg_mcast_joined retry 6600
>Jul 29 13:44:30 hayes-02 groupd[3148]: cpg_mcast_joined retry 6700
>Jul 29 13:44:30 hayes-02 groupd[3148]: cpg_mcast_joined retry 6800
>Jul 29 13:44:31 hayes-02 groupd[3148]: cpg_mcast_joined retry 6900
>Jul 29 13:44:31 hayes-02 groupd[3148]: cpg_mcast_joined retry 7000
>Jul 29 13:44:31 hayes-02 groupd[3148]: cpg_mcast_joined retry 7100
>Jul 29 13:44:31 hayes-02 groupd[3148]: cpg_mcast_joined retry 7200
>Jul 29 13:44:31 hayes-02 groupd[3148]: cpg_mcast_joined retry 7300
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] entering GATHER state from 0. 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] Creating commit token because I am the rep. 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] Saving state aru 3 high seq received 3 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] Storing new sequence id for ring 2a8 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] entering COMMIT state. 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] entering RECOVERY state. 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] position [0] member 10.15.89.136: 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] previous ring seq 676 rep 10.15.89.136 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] aru 3 high delivered 3 received flag 1 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] Did not need to originate any messages in recovery. 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] Sending initial ORF token 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] New Configuration: 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] Members Left: 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] Members Joined: 
>Jul 29 13:44:31 hayes-02 openais[3135]: [SYNC ] This node is within the primary component and will provide service. 
>Jul 29 13:44:31 hayes-02 openais[3135]: [TOTEM] entering OPERATIONAL state. 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] got nodejoin message 10.15.89.136 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CLM  ] got nodejoin message 10.15.89.136 
>Jul 29 13:44:31 hayes-02 openais[3135]: [CPG  ] got joinlist message from node 2

Actions: View

Attachments on bug 457107: 312922 | 312923 | 636951