Attachment 300249 Details for Bug 439196 – Another crash-bt

Another crash-bt

cla0-syslog-fsprocjam2.txt (text/plain), 22.67 KB, created by Janne Karhunen on 2008-04-03 13:54:27 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Janne Karhunen

Created: 2008-04-03 13:54:27 UTC

Size: 22.67 KB

patch

obsolete

>Mar 19 12:31:12 info CLA-0 drbdmonitor: device 2 state changed from Connected Secondary/Secondary UpToDate/UpToDate to Connected Primary/Secondary UpToDate/UpToDate
>Mar 19 12:31:12 info CLA-0 drbdmonitor: State to be checked Connected Primary UpToDate
>Mar 19 12:31:12 info CLA-0 drbdmonitor: Setting state for 2 from secondary  to primary 
>Mar 19 12:31:12 info CLA-0 drbdmonitor: State set
>Mar 19 12:31:12 info CLA-0 HASNodeAgent: attach_file_system: /opt/Nokia_BP/bin/attach_file_system /dev/drbd2 log
>Mar 19 12:31:12 info CLA-0 kernel: kjournald starting.  Commit interval 5 seconds
>Mar 19 12:31:12 warning CLA-0 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
>Mar 19 12:31:12 info CLA-0 kernel: EXT3 FS on drbd2, internal journal
>Mar 19 12:31:12 info CLA-0 kernel: EXT3-fs: mounted filesystem with ordered data mode.
>Mar 19 12:31:12 info CLA-0 attach_file_system: File system mount succeeded: /dev/drbd2 mounted on /var/mnt/local/log
>Mar 19 12:31:12 info CLA-0 HASNodeAgent: attach_file_system: /opt/Nokia_BP/bin/attach_file_system /dev/VG_62/backup backup
>Mar 19 12:31:12 info CLA-0 kernel: kjournald starting.  Commit interval 5 seconds
>Mar 19 12:31:12 warning CLA-0 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
>Mar 19 12:31:12 info CLA-0 kernel: EXT3 FS on dm-8, internal journal
>Mar 19 12:31:12 info CLA-0 kernel: EXT3-fs: mounted filesystem with ordered data mode.
>Mar 19 12:31:12 info CLA-0 attach_file_system: File system mount succeeded: /dev/VG_62/backup mounted on /var/mnt/local/backup
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: EXEC: /opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh start  active
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: ++ /opt/Nokia_BP/bin/fslistnodes -H ldap://localhost:49501
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: ++ wc -l
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + NODECOUNT=5
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + PRIMARY_CONFIG=/var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/etc/ldapfiles/fsPlatform.conf
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + SECONDARY_CONFIG=/opt/Nokia_BP/etc/ldapfiles/fsPlatformSlave.conf
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + LOCAL_STARTUP_CONFIG=/opt/Nokia_BP/etc/startupconfig.txt
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + STARTUP_CONFIG=/var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/etc/startupconfig.txt
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + STORED_CONFIGS=/var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/pmgmt/_active/
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + mo=/CLA-0/FSDirectoryServer
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + name=/opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + \'[\' 2 -ne 2 \']\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + operation=start
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + role=active
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + RPCNFSDCOUNT=32
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + RPCNFSDTAGFILE=/tmp/_rpc.nfsd.tag
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + RPCMOUNTDOPTIONS=\'-N 2 -p 190\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + STATDOPTIONS=\'-n Directory -p 191\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + RPCMOUNTDTAGFILE=/tmp/_rpc.mountd.tag
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + DSRTPRIO=20
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + chrt -rp 20 12070
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + log debug \'/CLA-0/FSDirectoryServer: /opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh start active\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + local prio=debug
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + shift
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + /usr/bin/logger -t \'/opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh[12070]\' -p user.debug \'/CLA-0/FSDirectoryServer: /opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh start active\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + case \"$operation\" in
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + do_start
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + case \"$role\" in
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + [[ -s /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/etc/startupconfig.txt ]]
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + mount -t rpc_pipefs sunrpc /var/lib/rpc_pipefs
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + LOGMNT=/var/mnt/local/log
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + SYSMNT=/var/mnt/local/sysimg
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + LOGSYS=/var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/log
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + grep -q /var/mnt/local/log /proc/mounts
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + mkdir -p /var/mnt/local/log/log/master
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + test -d /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/log
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + mv /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/log /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/log.sysimg
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + ln -nsf /var/mnt/local/log/log/master /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/log
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + touch /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/.logpartition_in_use
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + nokiabp=/opt/Nokia_BP_at_sysimg/
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + nokiabp_var=/var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + log info \'Making syslog and NFS symlinks to system image\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + local prio=info
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + shift
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + /usr/bin/logger -t \'/opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh[12070]\' -p user.info \'Making syslog and NFS symlinks to system image\'
>Mar 19 12:31:12 info CLA-0 /opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh[12070]: Making syslog and NFS symlinks to system image
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + mkdir -p /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var//log
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + rm -rf /var/log/master-syslog
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + rm -rf /var/log/master
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + ln -nsf /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var//log/syslog /var/log/master-syslog
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + ln -nsf /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var//log /var/log/master
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + rm -rf /etc/exports
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + ln -nsf /opt/Nokia_BP_at_sysimg//etc/exports /etc/exports
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + \'[\' 5 -gt 1 \']\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + log info \'Starting NFS NSM\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + local prio=info
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + shift
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + /usr/bin/logger -t \'/opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh[12070]\' -p user.info \'Starting NFS NSM\'
>Mar 19 12:31:12 info CLA-0 /opt/Nokia_BP/SS_BPUtils/script/RUControlDirectoryServer.sh[12070]: Starting NFS NSM
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: + \'[\' \'!\' \']\'
>Mar 19 12:31:12 info CLA-0 HASNodeAgent[9898]: INFO RUControlDirectoryServer.sh: ++ /sbin/pidof rpc.statd
>Mar 19 12:31:16 info CLA-0 HASClusterManag[11177]: INFO NodeTerminationObserver thread starting...
>Mar 19 12:31:16 info CLA-0 HASClusterManag[11177]: INFO Waiting for node /CLA-1 to go down...
>Mar 19 12:31:17 info CLA-0 HASClusterManag[11177]: INFO Connection with /CLA-1 () down : Peer process closed connection
>Mar 19 12:31:17 info CLA-0 HASClusterManag[11177]: INFO StartupController: HAS startup level of node /CLA-1 changed: 2 => 0.
>Mar 19 12:31:20 info CLA-0 OSMON: System Alert: NFS server is down (Directory:mount, error=\'RPC: Timed out\')
>Mar 19 12:31:37 err CLA-0 kernel: drbd6: PingAck did not arrive in time.
>Mar 19 12:31:37 info CLA-0 kernel: drbd6: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
>Mar 19 12:31:37 info CLA-0 kernel: drbd6: asender terminated
>Mar 19 12:31:37 err CLA-0 kernel: drbd6: short read expecting header on sock: r=-512
>Mar 19 12:31:37 warning CLA-0 kernel: drbd6: tl_clear()
>Mar 19 12:31:37 info CLA-0 kernel: drbd6: Connection closed
>Mar 19 12:31:37 info CLA-0 kernel: drbd6: Writing meta data super block now.
>Mar 19 12:31:37 info CLA-0 kernel: drbd6: conn( NetworkFailure -> Unconnected ) 
>Mar 19 12:31:37 info CLA-0 kernel: drbd6: receiver terminated
>Mar 19 12:31:37 info CLA-0 kernel: drbd6: receiver (re)started
>Mar 19 12:31:37 info CLA-0 kernel: drbd6: conn( Unconnected -> WFConnection ) 
>Mar 19 12:31:37 info CLA-0 HASClusterManag[11177]: INFO Node /CLA-1 down.
>Mar 19 12:31:37 info CLA-0 HASClusterManag[11177]: INFO Watchdog info for /CLA-1: Timer stopped Reserved PreTimeout:None TimerAction:No action pre:0 init:0 pres:0
>Mar 19 12:31:37 info CLA-0 HASClusterManag[11177]: INFO Issuing hardware reset to node CLA-1
>Mar 19 12:31:37 info CLA-0 drbdmonitor: device 6 state changed from Connected Secondary/Secondary UpToDate/UpToDate to WFConnection Secondary/Unknown UpToDate/DUnknown
>Mar 19 12:31:37 info CLA-0 drbdmonitor: State to be checked WFConnection Secondary UpToDate
>Mar 19 12:31:37 info CLA-0 drbdmonitor: Setting state for 6 from secondary  to secondary --drbd-alone
>Mar 19 12:31:37 info CLA-0 drbdmonitor: State set
>Mar 19 12:31:38 err CLA-0 kernel: drbd7: PingAck did not arrive in time.
>Mar 19 12:31:38 info CLA-0 kernel: drbd7: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
>Mar 19 12:31:38 info CLA-0 kernel: drbd7: Creating new current UUID
>Mar 19 12:31:38 info CLA-0 kernel: drbd7: asender terminated
>Mar 19 12:31:38 err CLA-0 kernel: drbd7: short read expecting header on sock: r=-512
>Mar 19 12:31:38 warning CLA-0 kernel: drbd7: tl_clear()
>Mar 19 12:31:38 info CLA-0 kernel: drbd7: Connection closed
>Mar 19 12:31:38 info CLA-0 kernel: drbd7: Writing meta data super block now.
>Mar 19 12:31:38 info CLA-0 drbdadm: outdating peer IP:CLA-1 device:/dev/drbd7
>Mar 19 12:31:38 info CLA-0 drbdmonitor: device 7 state changed from Connected Primary/Secondary UpToDate/UpToDate to NetworkFailure Primary/Unknown UpToDate/DUnknown
>Mar 19 12:31:38 info CLA-0 drbdmonitor: State to be checked NetworkFailure Primary UpToDate
>Mar 19 12:31:38 info CLA-0 drbdmonitor: Setting state for 7 from primary  to primary --drbd-alone
>Mar 19 12:31:38 info CLA-0 drbdmonitor: State set
>Mar 19 12:31:38 err CLA-0 kernel: drbd5: PingAck did not arrive in time.
>Mar 19 12:31:38 info CLA-0 kernel: drbd5: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
>Mar 19 12:31:38 info CLA-0 kernel: drbd5: Creating new current UUID
>Mar 19 12:31:38 info CLA-0 kernel: drbd5: asender terminated
>Mar 19 12:31:38 err CLA-0 kernel: drbd5: short read expecting header on sock: r=-512
>Mar 19 12:31:38 warning CLA-0 kernel: drbd5: tl_clear()
>Mar 19 12:31:38 info CLA-0 kernel: drbd5: Connection closed
>Mar 19 12:31:38 info CLA-0 kernel: drbd5: Writing meta data super block now.
>Mar 19 12:31:38 info CLA-0 drbdadm: outdating peer IP:CLA-1 device:/dev/drbd5
>Mar 19 12:31:38 info CLA-0 hwmd: Node \'CLA-1\' has been reset within 1.58 second(s)
>Mar 19 12:31:39 err CLA-0 kernel: drbd4: PingAck did not arrive in time.
>Mar 19 12:31:39 info CLA-0 kernel: drbd4: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
>Mar 19 12:31:39 info CLA-0 kernel: drbd4: Creating new current UUID
>Mar 19 12:31:39 info CLA-0 kernel: drbd4: asender terminated
>Mar 19 12:31:39 err CLA-0 kernel: drbd4: short read expecting header on sock: r=-512
>Mar 19 12:31:39 warning CLA-0 kernel: drbd4: tl_clear()
>Mar 19 12:31:39 info CLA-0 kernel: drbd4: Connection closed
>Mar 19 12:31:39 info CLA-0 kernel: drbd4: Writing meta data super block now.
>Mar 19 12:31:39 info CLA-0 drbdadm: outdating peer IP:CLA-1 device:/dev/drbd4
>Mar 19 12:31:39 info CLA-0 drbdmonitor: device 4 state changed from Connected Primary/Secondary UpToDate/UpToDate to NetworkFailure Primary/Unknown UpToDate/DUnknown
>Mar 19 12:31:39 info CLA-0 drbdmonitor: State to be checked NetworkFailure Primary UpToDate
>Mar 19 12:31:39 info CLA-0 drbdmonitor: Setting state for 4 from primary  to primary --drbd-alone
>Mar 19 12:31:39 info CLA-0 drbdmonitor: State set
>Mar 19 12:31:39 info CLA-0 drbdmonitor: device 5 state changed from Connected Primary/Secondary UpToDate/UpToDate to NetworkFailure Primary/Unknown UpToDate/DUnknown
>Mar 19 12:31:39 info CLA-0 drbdmonitor: State to be checked NetworkFailure Primary UpToDate
>Mar 19 12:31:39 info CLA-0 drbdmonitor: Setting state for 5 from primary  to primary --drbd-alone
>Mar 19 12:31:39 info CLA-0 drbdmonitor: State set
>Mar 19 12:31:43 info CLA-0 drbdadm: outdate failed, assuming that the peer is down
>Mar 19 12:31:43 info CLA-0 kernel: drbd7: outdate-peer helper returned 5 
>Mar 19 12:31:43 info CLA-0 kernel: drbd7: pdsk( DUnknown -> Outdated ) 
>Mar 19 12:31:43 info CLA-0 kernel: drbd7: conn( NetworkFailure -> Unconnected ) 
>Mar 19 12:31:43 info CLA-0 kernel: drbd7: receiver terminated
>Mar 19 12:31:43 info CLA-0 kernel: drbd7: receiver (re)started
>Mar 19 12:31:43 info CLA-0 kernel: drbd7: conn( Unconnected -> WFConnection ) 
>Mar 19 12:31:43 err CLA-0 kernel: drbd1: PingAck did not arrive in time.
>Mar 19 12:31:43 info CLA-0 kernel: drbd1: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
>Mar 19 12:31:43 info CLA-0 kernel: drbd1: Creating new current UUID
>Mar 19 12:31:43 info CLA-0 kernel: drbd1: asender terminated
>Mar 19 12:31:43 err CLA-0 kernel: drbd1: short read expecting header on sock: r=-512
>Mar 19 12:31:43 warning CLA-0 kernel: drbd1: tl_clear()
>Mar 19 12:31:43 info CLA-0 kernel: drbd1: Connection closed
>Mar 19 12:31:43 info CLA-0 kernel: drbd1: Writing meta data super block now.
>Mar 19 12:31:43 info CLA-0 drbdadm: outdating peer IP:CLA-1 device:/dev/drbd1
>Mar 19 12:31:43 info CLA-0 drbdadm: outdate failed, assuming that the peer is down
>Mar 19 12:31:43 info CLA-0 kernel: drbd5: outdate-peer helper returned 5 
>Mar 19 12:31:43 info CLA-0 kernel: drbd5: pdsk( DUnknown -> Outdated ) 
>Mar 19 12:31:43 info CLA-0 kernel: drbd5: conn( NetworkFailure -> Unconnected ) 
>Mar 19 12:31:43 info CLA-0 kernel: drbd5: receiver terminated
>Mar 19 12:31:43 info CLA-0 kernel: drbd5: receiver (re)started
>Mar 19 12:31:43 info CLA-0 kernel: drbd5: conn( Unconnected -> WFConnection ) 
>Mar 19 12:31:43 info CLA-0 drbdmonitor: device 1 state changed from Connected Primary/Secondary UpToDate/UpToDate to NetworkFailure Primary/Unknown UpToDate/DUnknown
>Mar 19 12:31:43 info CLA-0 drbdmonitor: State to be checked NetworkFailure Primary UpToDate
>Mar 19 12:31:43 info CLA-0 drbdmonitor: Setting state for 1 from primary  to primary --drbd-alone
>Mar 19 12:31:43 info CLA-0 drbdmonitor: State set
>Mar 19 12:31:43 info CLA-0 drbdmonitor: device 5 state changed from NetworkFailure Primary/Unknown UpToDate/DUnknown to WFConnection Primary/Unknown UpToDate/Outdated
>Mar 19 12:31:43 info CLA-0 drbdmonitor: State to be checked WFConnection Primary UpToDate
>Mar 19 12:31:43 info CLA-0 drbdmonitor: Setting state for 5 to primary --drbd-alone skipped, state not changed
>Mar 19 12:31:43 info CLA-0 drbdmonitor: device 7 state changed from NetworkFailure Primary/Unknown UpToDate/DUnknown to WFConnection Primary/Unknown UpToDate/Outdated
>Mar 19 12:31:43 info CLA-0 drbdmonitor: State to be checked WFConnection Primary UpToDate
>Mar 19 12:31:43 info CLA-0 drbdmonitor: Setting state for 7 to primary --drbd-alone skipped, state not changed
>Mar 19 12:31:44 info CLA-0 drbdadm: outdate failed, assuming that the peer is down
>Mar 19 12:31:44 info CLA-0 kernel: drbd4: outdate-peer helper returned 5 
>Mar 19 12:31:44 info CLA-0 kernel: drbd4: pdsk( DUnknown -> Outdated ) 
>Mar 19 12:31:44 info CLA-0 kernel: drbd4: conn( NetworkFailure -> Unconnected ) 
>Mar 19 12:31:44 info CLA-0 kernel: drbd4: receiver terminated
>Mar 19 12:31:44 info CLA-0 kernel: drbd4: receiver (re)started
>Mar 19 12:31:44 info CLA-0 kernel: drbd4: conn( Unconnected -> WFConnection ) 
>Mar 19 12:31:44 err CLA-0 kernel: drbd0: PingAck did not arrive in time.
>Mar 19 12:31:44 info CLA-0 kernel: drbd0: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
>Mar 19 12:31:44 info CLA-0 kernel: drbd0: Creating new current UUID
>Mar 19 12:31:44 info CLA-0 kernel: drbd0: asender terminated
>Mar 19 12:31:44 err CLA-0 kernel: drbd0: short read expecting header on sock: r=-512
>Mar 19 12:31:44 warning CLA-0 kernel: drbd0: tl_clear()
>Mar 19 12:31:44 info CLA-0 kernel: drbd0: Connection closed
>Mar 19 12:31:44 info CLA-0 kernel: drbd0: Writing meta data super block now.
>Mar 19 12:31:44 err CLA-0 kernel: drbd2: PingAck did not arrive in time.
>Mar 19 12:31:44 info CLA-0 kernel: drbd2: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
>Mar 19 12:31:44 info CLA-0 kernel: drbd2: Creating new current UUID
>Mar 19 12:31:44 info CLA-0 kernel: drbd2: asender terminated
>Mar 19 12:31:44 err CLA-0 kernel: drbd2: short read expecting header on sock: r=-512
>Mar 19 12:31:44 warning CLA-0 kernel: drbd2: tl_clear()
>Mar 19 12:31:44 info CLA-0 kernel: drbd2: Connection closed
>Mar 19 12:31:44 info CLA-0 kernel: drbd2: Writing meta data super block now.
>Mar 19 12:31:44 info CLA-0 drbdadm: outdating peer IP:CLA-1 device:/dev/drbd0
>Mar 19 12:31:44 info CLA-0 drbdmonitor: device 0 state changed from Connected Primary/Secondary UpToDate/UpToDate to NetworkFailure Primary/Unknown UpToDate/DUnknown
>Mar 19 12:31:44 info CLA-0 drbdmonitor: State to be checked NetworkFailure Primary UpToDate
>Mar 19 12:31:44 info CLA-0 drbdmonitor: Setting state for 0 from primary  to primary --drbd-alone
>Mar 19 12:31:44 info CLA-0 drbdadm: outdating peer IP:CLA-1 device:/dev/drbd2
>Mar 19 12:31:46 info CLA-0 drbdmonitor: command \"/opt/Nokia_BP/bin/fsdrbdcli -p /dev/drbd0 --drbd-status primary --drbd-alone --drbd-node /CLA-0\" failed, retrying forever
>Mar 19 12:31:48 warning CLA-0 kernel: drbd7: BUG! md_sync_timer expired! Worker calls drbd_md_sync().
>Mar 19 12:31:48 info CLA-0 kernel: drbd7: Writing meta data super block now.
>Mar 19 12:31:48 info CLA-0 drbdadm: outdate failed, assuming that the peer is down
>Mar 19 12:31:48 info CLA-0 kernel: drbd1: outdate-peer helper returned 5 
>Mar 19 12:31:48 info CLA-0 kernel: drbd1: pdsk( DUnknown -> Outdated ) 
>Mar 19 12:31:48 info CLA-0 kernel: drbd1: conn( NetworkFailure -> Unconnected ) 
>Mar 19 12:31:48 info CLA-0 kernel: drbd1: receiver terminated
>Mar 19 12:31:48 info CLA-0 kernel: drbd1: receiver (re)started
>Mar 19 12:31:48 info CLA-0 kernel: drbd1: conn( Unconnected -> WFConnection ) 
>Mar 19 12:31:48 warning CLA-0 kernel: drbd5: BUG! md_sync_timer expired! Worker calls drbd_md_sync().
>Mar 19 12:31:48 info CLA-0 kernel: drbd5: Writing meta data super block now.
>Mar 19 12:31:49 warning CLA-0 kernel: drbd4: BUG! md_sync_timer expired! Worker calls drbd_md_sync().
>Mar 19 12:31:49 info CLA-0 kernel: drbd4: Writing meta data super block now.
>Mar 19 12:31:49 info CLA-0 drbdadm: outdate failed, assuming that the peer is down
>Mar 19 12:31:49 info CLA-0 kernel: drbd0: outdate-peer helper returned 5 
>Mar 19 12:31:49 info CLA-0 kernel: drbd0: pdsk( DUnknown -> Outdated ) 
>Mar 19 12:31:49 info CLA-0 kernel: drbd0: conn( NetworkFailure -> Unconnected ) 
>Mar 19 12:31:49 info CLA-0 kernel: drbd0: receiver terminated
>Mar 19 12:31:49 info CLA-0 kernel: drbd0: receiver (re)started
>Mar 19 12:31:49 info CLA-0 kernel: drbd0: conn( Unconnected -> WFConnection ) 
>Mar 19 12:31:49 info CLA-0 drbdadm: outdate failed, assuming that the peer is down
>Mar 19 12:31:49 info CLA-0 kernel: drbd2: outdate-peer helper returned 5 
>Mar 19 12:31:49 info CLA-0 kernel: drbd2: pdsk( DUnknown -> Outdated ) 
>Mar 19 12:31:49 info CLA-0 kernel: drbd2: conn( NetworkFailure -> Unconnected ) 
>Mar 19 12:31:49 info CLA-0 kernel: drbd2: receiver terminated
>Mar 19 12:31:49 info CLA-0 kernel: drbd2: receiver (re)started
>Mar 19 12:31:49 info CLA-0 kernel: drbd2: conn( Unconnected -> WFConnection ) 
>Mar 19 12:31:53 warning CLA-0 kernel: drbd1: BUG! md_sync_timer expired! Worker calls drbd_md_sync().
>Mar 19 12:31:53 info CLA-0 kernel: drbd1: Writing meta data super block now.
>Mar 19 12:31:54 warning CLA-0 kernel: drbd0: BUG! md_sync_timer expired! Worker calls drbd_md_sync().
>Mar 19 12:31:54 info CLA-0 kernel: drbd0: Writing meta data super block now.
>Mar 19 12:31:54 warning CLA-0 kernel: drbd2: BUG! md_sync_timer expired! Worker calls drbd_md_sync().
>Mar 19 12:31:54 info CLA-0 kernel: drbd2: Writing meta data super block now.
>Mar 19 12:32:01 info CLA-0 crond[12174]: (root) CMD (/etc/init.d/hwmd check >/dev/null 2>&1)
>Mar 19 12:32:01 info CLA-0 crond[12175]: (root) CMD (/opt/Nokia_BP/bin/sel >/dev/null 2>&1)

Actions: View

Attachments on bug 439196: 299335 | 300059 | 300244 | 300249