Attachment 308450 Details for Bug 450161 – log from hayes-02

log from hayes-02

hayes-02.log (text/plain), 11.62 KB, created by Corey Marthaler on 2008-06-05 16:29:13 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2008-06-05 16:29:13 UTC

Size: 11.62 KB

patch

obsolete

>Jun  5 10:40:12 hayes-02 openais[14661]: [TOTEM] The token was lost in the OPERATIONAL state. 
>Jun  5 10:40:12 hayes-02 openais[14661]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
>Jun  5 10:40:12 hayes-02 openais[14661]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jun  5 10:40:12 hayes-02 openais[14661]: [TOTEM] entering GATHER state from 2. 
>Jun  5 10:40:15 hayes-02 xinetd[2722]: START: qarsh pid=14969 from=10.15.80.47
>Jun  5 10:40:15 hayes-02 qarshd[14969]: Talking to peer 10.15.80.47:53888
>Jun  5 10:40:15 hayes-02 qarshd[14969]: Running cmdline: cman_tool nodes 
>Jun  5 10:40:15 hayes-02 xinetd[2722]: EXIT: qarsh status=0 pid=14969 duration=0(sec)
>Jun  5 10:40:15 hayes-02 xinetd[2722]: START: qarsh pid=14971 from=10.15.80.47
>Jun  5 10:40:15 hayes-02 qarshd[14971]: Talking to peer 10.15.80.47:53889
>Jun  5 10:40:15 hayes-02 qarshd[14971]: Running cmdline: cman_tool nodes 
>Jun  5 10:40:15 hayes-02 xinetd[2722]: EXIT: qarsh status=0 pid=14971 duration=0(sec)
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] entering GATHER state from 0. 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] Creating commit token because I am the rep. 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] Saving state aru e2 high seq received e2 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] Storing new sequence id for ring 1f4 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] entering COMMIT state. 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] entering RECOVERY state. 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] position [0] member 10.15.89.136: 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] previous ring seq 496 rep 10.15.89.135 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] aru e2 high delivered e2 received flag 1 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] position [1] member 10.15.89.137: 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] previous ring seq 496 rep 10.15.89.135 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] aru e2 high delivered e2 received flag 1 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] Did not need to originate any messages in recovery. 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] Sending initial ORF token 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] New Configuration: 
>Jun  5 10:40:17 hayes-02 kernel: dlm: closing connection to node 1
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] Members Left: 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] Members Joined: 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] New Configuration: 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] Members Left: 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] Members Joined: 
>Jun  5 10:40:17 hayes-02 openais[14661]: [SYNC ] This node is within the primary component and will provide service. 
>Jun  5 10:40:17 hayes-02 openais[14661]: [TOTEM] entering OPERATIONAL state. 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] got nodejoin message 10.15.89.136 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CLM  ] got nodejoin message 10.15.89.137 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CPG  ] got joinlist message from node 3 
>Jun  5 10:40:17 hayes-02 openais[14661]: [CPG  ] got joinlist message from node 2 
>Jun  5 10:40:20 hayes-02 xinetd[2722]: START: qarsh pid=14973 from=10.15.80.47
>Jun  5 10:40:20 hayes-02 qarshd[14973]: Talking to peer 10.15.80.47:53913
>Jun  5 10:40:20 hayes-02 qarshd[14973]: Running cmdline: cman_tool nodes 
>Jun  5 10:40:20 hayes-02 xinetd[2722]: EXIT: qarsh status=0 pid=14973 duration=0(sec)
>Jun  5 10:40:47 hayes-02 fenced[14679]: hayes-01 not a cluster member after 30 sec post_fail_delay
>Jun  5 10:40:47 hayes-02 fenced[14679]: fencing node "hayes-01"
>Jun  5 10:41:09 hayes-02 fenced[14679]: fence "hayes-01" success
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES0.1: jid=0: Trying to acquire journal lock...
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES1.1: jid=0: Trying to acquire journal lock...
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES0.1: jid=0: Looking at journal...
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES0.1: jid=0: Acquiring the transaction lock...
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES0.1: jid=0: Replaying journal...
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES0.1: jid=0: Replayed 0 of 2 blocks
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES0.1: jid=0: replays = 0, skips = 2, sames = 0
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES0.1: jid=0: Journal replayed in 1s
>Jun  5 10:41:09 hayes-02 kernel: GFS: fsid=HAYES:HAYES0.1: jid=0: Done
>Jun  5 10:41:10 hayes-02 kernel: GFS: fsid=HAYES:HAYES1.1: jid=0: Looking at journal...
>Jun  5 10:41:10 hayes-02 kernel: GFS: fsid=HAYES:HAYES1.1: jid=0: Acquiring the transaction lock...
>Jun  5 10:41:10 hayes-02 kernel: GFS: fsid=HAYES:HAYES1.1: jid=0: Replaying journal...
>Jun  5 10:41:10 hayes-02 kernel: GFS: fsid=HAYES:HAYES1.1: jid=0: Replayed 0 of 8 blocks
>Jun  5 10:41:10 hayes-02 kernel: GFS: fsid=HAYES:HAYES1.1: jid=0: replays = 0, skips = 8, sames = 0
>Jun  5 10:41:10 hayes-02 kernel: GFS: fsid=HAYES:HAYES1.1: jid=0: Journal replayed in 1s
>Jun  5 10:41:10 hayes-02 kernel: GFS: fsid=HAYES:HAYES1.1: jid=0: Done
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] entering GATHER state from 11. 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] Saving state aru 3a high seq received 3a 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] Storing new sequence id for ring 1f8 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] entering COMMIT state. 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] entering RECOVERY state. 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] position [0] member 10.15.89.135: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] previous ring seq 500 rep 10.15.89.135 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] aru a high delivered a received flag 1 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] position [1] member 10.15.89.136: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] previous ring seq 500 rep 10.15.89.136 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] aru 3a high delivered 3a received flag 1 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] position [2] member 10.15.89.137: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] previous ring seq 500 rep 10.15.89.136 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] aru 3a high delivered 3a received flag 1 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] Did not need to originate any messages in recovery. 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] New Configuration: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] Members Left: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] Members Joined: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] New Configuration: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] Members Left: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] Members Joined: 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jun  5 10:43:21 hayes-02 openais[14661]: [SYNC ] This node is within the primary component and will provide service. 
>Jun  5 10:43:21 hayes-02 openais[14661]: [TOTEM] entering OPERATIONAL state. 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] got nodejoin message 10.15.89.135 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] got nodejoin message 10.15.89.136 
>Jun  5 10:43:21 hayes-02 openais[14661]: [CLM  ] got nodejoin message 10.15.89.137 
>Jun  5 10:43:31 hayes-02 openais[14661]: [TOTEM] The token was lost in the OPERATIONAL state. 
>Jun  5 10:43:31 hayes-02 openais[14661]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
>Jun  5 10:43:31 hayes-02 openais[14661]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jun  5 10:43:31 hayes-02 openais[14661]: [TOTEM] entering GATHER state from 2. 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] entering GATHER state from 11. 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] Creating commit token because I am the rep. 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] Saving state aru 17 high seq received 17 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] Storing new sequence id for ring 1fc 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] entering COMMIT state. 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] entering RECOVERY state. 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] position [0] member 10.15.89.136: 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] previous ring seq 504 rep 10.15.89.135 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] aru 17 high delivered 17 received flag 1 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] position [1] member 10.15.89.137: 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] previous ring seq 504 rep 10.15.89.135 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] aru 17 high delivered 17 received flag 1 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] Did not need to originate any messages in recovery. 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] Sending initial ORF token 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] New Configuration: 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] Members Left: 
>Jun  5 10:43:36 hayes-02 kernel: dlm: closing connection to node 1
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] Members Joined: 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] New Configuration: 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] Members Left: 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] Members Joined: 
>Jun  5 10:43:36 hayes-02 openais[14661]: [SYNC ] This node is within the primary component and will provide service. 
>Jun  5 10:43:36 hayes-02 openais[14661]: [TOTEM] entering OPERATIONAL state. 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] got nodejoin message 10.15.89.136 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CLM  ] got nodejoin message 10.15.89.137 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CPG  ] got joinlist message from node 3 
>Jun  5 10:43:36 hayes-02 openais[14661]: [CPG  ] got joinlist message from node 2 
>Jun  5 11:11:37 hayes-02 qarshd[14946]: Sending child 14948 signal 2
>Jun  5 11:11:37 hayes-02 qarshd[14947]: Sending child 14949 signal 2
>Jun  5 11:11:37 hayes-02 xinetd[2722]: EXIT: qarsh status=0 pid=14946 duration=1969(sec)
>Jun  5 11:11:37 hayes-02 xinetd[2722]: EXIT: qarsh status=0 pid=14947 duration=1969(sec)

Actions: View

Attachments on bug 450161: 308449 | 308450 | 308452