Attachment 151204 Details for Bug 234410 – /var/log/messages and console output from ppc cluster

/var/log/messages and console output from ppc cluster

ppc-logs (text/plain), 5.82 KB, created by Nate Straz on 2007-03-29 15:07:01 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Nate Straz

Created: 2007-03-29 15:07:01 UTC

Size: 5.82 KB

patch

obsolete

>Mar 29 09:33:15 basic ccsd[4086]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.7.3 
>Mar 29 09:33:15 basic ccsd[4086]: Initial status:: Inquorate 
>Mar 29 09:33:46 basic kernel: CMAN: forming a new cluster
>Mar 29 09:33:48 basic kernel: CMAN: quorum regained, resuming activity
>Mar 29 09:33:48 basic ccsd[4086]: Cluster is quorate.  Allowing connections. 
>Mar 29 09:33:48 basic cman: startup succeeded
>Mar 29 09:38:40 basic fenced: startup succeeded
>Mar 29 09:58:32 basic clvmd: clvmd startup timed out
>Mar 29 09:58:32 basic clvmd: clvmd startup failed
>Mar 29 09:58:36 basic kernel: CMAN: node newport has been removed from the cluster : No response to messages
>Mar 29 09:58:42 basic kernel: CMAN: removing node doral from the cluster : No response to messages
>Mar 29 09:58:48 basic kernel: CMAN: removing node kent from the cluster : No response to messages
>Mar 29 09:59:06 basic kernel: CMAN: quorum lost, blocking activity
>Mar 29 09:59:06 basic kernel: CMAN: removing node doral from the cluster : No response to messages
>Mar 29 09:59:06 basic kernel: CMAN: node basic has been removed from the cluster : No response to messages
>Mar 29 09:59:06 basic kernel: CMAN: killed by NODEDOWN message
>Mar 29 09:59:06 basic kernel: CMAN: we are leaving the cluster. No response to messages
>Mar 29 09:59:06 basic kernel: WARNING: dlm_emergency_shutdown
>Mar 29 09:59:06 basic ccsd[4086]: Cluster manager shutdown.  Attemping to reconnect... 
>Mar 29 09:59:06 basic fenced[4188]: process_events: service get event failed
>
>Mar 29 09:33:16 doral ccsd[7239]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.7.3 
>Mar 29 09:33:16 doral ccsd[7239]: Initial status:: Inquorate 
>Mar 29 09:33:47 doral kernel: CMAN: sending membership request
>Mar 29 09:33:52 doral kernel: CMAN: sending membership request
>Mar 29 09:33:53 doral kernel: CMAN: got node kent
>Mar 29 09:33:53 doral kernel: CMAN: got node newport
>Mar 29 09:33:53 doral kernel: CMAN: got node basic
>Mar 29 09:33:53 doral kernel: CMAN: quorum regained, resuming activity
>Mar 29 09:33:53 doral ccsd[7239]: Cluster is quorate.  Allowing connections.
>Mar 29 09:33:53 doral cman: startup succeeded
>Mar 29 09:38:41 doral fenced: startup succeeded
>Mar 29 09:58:29 doral kernel: CMAN: node newport has been removed from the cluster : No response to messages
>Mar 29 09:58:32 doral clvmd: clvmd startup timed out
>Mar 29 09:58:32 doral clvmd: clvmd startup failed
>Mar 29 09:58:35 doral kernel: CMAN: node doral has been removed from the cluster: No response to messages
>Mar 29 09:58:35 doral kernel: CMAN: killed by NODEDOWN message
>Mar 29 09:58:35 doral kernel: CMAN: we are leaving the cluster. No response to messages
>Mar 29 09:58:35 doral kernel: WARNING: dlm_emergency_shutdown
>Mar 29 09:58:35 doral kernel: SM: send_nodeid_message error -107 to 3
>Mar 29 09:58:35 doral kernel: SM: send_nodeid_message error -107 to 1
>Mar 29 09:58:37 doral kernel: SM: send_broadcast_message error -107
>Mar 29 09:59:09 doral last message repeated 16 times
>Mar 29 10:00:11 doral last message repeated 31 times
>Mar 29 10:01:13 doral last message repeated 31 times
>
>Mar 29 09:33:16 kent ccsd[7294]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.7.3 
>Mar 29 09:33:16 kent ccsd[7294]: Initial status:: Inquorate 
>Mar 29 09:33:47 kent kernel: CMAN: sending membership request
>Mar 29 09:33:49 kent kernel: CMAN: got node newport
>Mar 29 09:33:49 kent kernel: CMAN: got node basic
>Mar 29 09:33:49 kent kernel: CMAN: quorum regained, resuming activity
>Mar 29 09:33:49 kent ccsd[7294]: Cluster is quorate.  Allowing connections. 
>Mar 29 09:33:49 kent cman: startup succeeded
>Mar 29 09:38:45 kent fenced: startup succeeded
>Mar 29 09:58:23 kent kernel: CMAN: removing node newport from the cluster : No response to messages
>Mar 29 09:58:32 kent clvmd: clvmd startup timed out
>Mar 29 09:58:32 kent clvmd: clvmd startup failed
>Mar 29 09:58:35 kent kernel: CMAN: removing node doral from the cluster : No response to messages
>Mar 29 09:58:37 kent kernel: CMAN: removing node basic from the cluster : No response to messages
>Mar 29 09:58:49 kent kernel: CMAN: quorum lost, blocking activity
>Mar 29 09:58:53 kent kernel: CMAN: node kent has been removed from the cluster : No response to messages
>Mar 29 09:58:53 kent kernel: CMAN: killed by NODEDOWN message
>Mar 29 09:58:53 kent kernel: CMAN: we are leaving the cluster. No response to messages
>Mar 29 09:58:53 kent kernel: WARNING: dlm_emergency_shutdown
>
>Mar 29 09:33:17 newport ccsd[7236]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.7.3 
>Mar 29 09:33:17 newport ccsd[7236]: Initial status:: Inquorate 
>Mar 29 09:33:47 newport kernel: CMAN: sending membership request
>Mar 29 09:33:47 newport kernel: CMAN: got node basic
>Mar 29 09:33:49 newport kernel: CMAN: quorum regained, resuming activity
>Mar 29 09:33:49 newport ccsd[7236]: Cluster is quorate.  Allowing connections. 
>Mar 29 09:33:49 newport cman: startup succeeded
>Mar 29 09:38:41 newport fenced: startup succeeded
>Mar 29 09:57:36 newport dhclient: DHCPREQUEST on eth0 to 10.15.89.100 port 67
>Mar 29 09:57:36 newport dhclient: DHCPACK from 10.15.89.100
>Mar 29 09:57:36 newport dhclient: bound to 10.15.89.60 -- renewal in 9502 seconds.
>Mar 29 09:58:29 newport kernel: CMAN: removing node doral from the cluster : No response to messages
>Mar 29 09:58:32 newport clvmd: clvmd startup timed out
>Mar 29 09:58:32 newport clvmd: clvmd startup failed
>Mar 29 09:58:34 newport clvmd: Cluster LVM daemon started - connected to CMAN
>Mar 29 09:58:35 newport kernel: CMAN: removing node kent from the cluster : No response to messages
>Mar 29 09:58:53 newport kernel: CMAN: removing node basic from the cluster : No response to messages
>
>On newport's console:
>
>SM:  Assertion failed on line 52 of file /builddir/build/BUILD/cman-kernel-2.6.9-49/up/src/sm_misc.c
>SM:  assertion:  "!error"
>SM:  time = 4352704632
>error = -1, nodeid = 4294967294
>
>Kernel panic - not syncing: SM:  Record message above and reboot.

Actions: View

Attachments on bug 234410: 151204