Attachment 293000 Details for Bug 430296 – log from hayes-03

log from hayes-03

hayes-03.log (text/plain), 49.84 KB, created by Corey Marthaler on 2008-01-25 21:13:36 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2008-01-25 21:13:36 UTC

Size: 49.84 KB

patch

obsolete

>Jan 25 13:38:57 hayes-03 kernel: eth0: no IPv6 routers present
>Jan 25 13:39:00 hayes-03 kernel: eth1: no IPv6 routers present
>Jan 25 13:45:48 hayes-03 kernel: DLM (built Jan 19 2008 22:04:07) installed
>Jan 25 13:45:48 hayes-03 kernel: GFS2 (built Jan 19 2008 22:04:52) installed
>Jan 25 13:45:48 hayes-03 kernel: Lock_DLM (built Jan 19 2008 22:05:05) installed
>Jan 25 13:45:48 hayes-03 ccsd[3775]: Starting ccsd 2.0.79: 
>Jan 25 13:45:48 hayes-03 ccsd[3775]:  Built: Jan 21 2008 15:39:03 
>Jan 25 13:45:48 hayes-03 ccsd[3775]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved. 
>Jan 25 13:45:48 hayes-03 ccsd[3775]: cluster.conf (cluster name = HAYES, version = 2) found. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] AIS Executive Service: started and ready to provide service. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Using default multicast address of 239.192.8.209 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_cpg loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_cfg loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais configuration service' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_msg loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais message service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_lck loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_evt loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais event service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_ckpt loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_amf loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_clm loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_evs loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] openais component openais_cman loaded. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] send threads (0 threads) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] RRP token expired timeout (495 ms) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] RRP token problem counter (2000 ms) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] RRP threshold (10 problem count) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] RRP mode set to none. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] heartbeat_failures_allowed (0) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] max_network_delay (50 ms) 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] The network interface [10.15.89.137] is now up. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Created or loaded sequence id 1084.10.15.89.137 for this ring. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] entering GATHER state from 15. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais event service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais message service B.01.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais configuration service' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CMAN ] CMAN 2.0.79 (built Jan 21 2008 15:39:11) started 
>Jan 25 13:46:04 hayes-03 openais[3790]: [SYNC ] Not using a virtual synchrony filter. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Creating commit token because I am the rep. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Saving state aru 0 high seq received 0 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Storing new sequence id for ring 440 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] entering COMMIT state. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] entering RECOVERY state. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] position [0] member 10.15.89.137: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] previous ring seq 1084 rep 10.15.89.137 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] aru 0 high delivered 0 received flag 1 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Did not need to originate any messages in recovery. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Sending initial ORF token 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 13:46:04 hayes-03 openais[3790]: [SYNC ] This node is within the primary component and will provide service. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] entering OPERATIONAL state. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.137 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] entering GATHER state from 11. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Saving state aru a high seq received a 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Storing new sequence id for ring 448 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] entering COMMIT state. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] entering RECOVERY state. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] position [0] member 10.15.89.135: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] previous ring seq 1092 rep 10.15.89.135 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] aru 17 high delivered 17 received flag 1 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] position [1] member 10.15.89.136: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] previous ring seq 1092 rep 10.15.89.135 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] aru 17 high delivered 17 received flag 1 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] position [2] member 10.15.89.137: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] previous ring seq 1088 rep 10.15.89.137 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] aru a high delivered 9 received flag 1 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] Did not need to originate any messages in recovery. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 13:46:04 hayes-03 openais[3790]: [SYNC ] This node is within the primary component and will provide service. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [TOTEM] entering OPERATIONAL state. 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CMAN ] quorum regained, resuming activity 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.135 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.136 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.137 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 1 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 2 
>Jan 25 13:46:04 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 3 
>Jan 25 13:46:04 hayes-03 ccsd[3775]: Cluster is not quorate.  Refusing connection. 
>Jan 25 13:46:04 hayes-03 ccsd[3775]: Error while processing connect: Connection refused 
>Jan 25 13:46:04 hayes-03 ccsd[3775]: Cluster is not quorate.  Refusing connection. 
>Jan 25 13:46:04 hayes-03 ccsd[3775]: Error while processing connect: Connection refused 
>Jan 25 13:46:04 hayes-03 ccsd[3775]: Initial status:: Quorate 
>Jan 25 13:47:03 hayes-03 kernel: aoe: aoe_init: AoE v22 initialised.
>Jan 25 13:47:03 hayes-03 kernel: aoe: 1.1 isbusy
>Jan 25 13:47:03 hayes-03 kernel: aoe: 0030488d63d3 e1.1 v4467 has 19046937725 sectors
>Jan 25 13:47:03 hayes-03 kernel:  etherd/e1.1: p1 p2 p3 p4 p5 p6 p7
>Jan 25 13:47:03 hayes-03 kernel: aoe: 1.1 isbusy
>Jan 25 13:47:03 hayes-03 last message repeated 2 times
>Jan 25 13:47:19 hayes-03 kernel: ACPI: PCI interrupt for device 0000:06:00.0 disabled
>Jan 25 13:47:22 hayes-03 kernel: QLogic Fibre Channel HBA Driver
>Jan 25 13:47:22 hayes-03 kernel: PCI: Enabling device 0000:06:00.0 (0140 -> 0143)
>Jan 25 13:47:22 hayes-03 kernel: ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 35 (level, low) -> IRQ 193
>Jan 25 13:47:22 hayes-03 kernel: qla2xxx 0000:06:00.0: Found an ISP2432, irq 193, iobase 0xffffc20000020000
>Jan 25 13:47:22 hayes-03 kernel: qla2xxx 0000:06:00.0: Configuring PCI space...
>Jan 25 13:47:22 hayes-03 kernel: PCI: Setting latency timer of device 0000:06:00.0 to 64
>Jan 25 13:47:22 hayes-03 kernel: qla2xxx 0000:06:00.0: Configure NVRAM parameters...
>Jan 25 13:47:22 hayes-03 kernel: qla2xxx 0000:06:00.0: Verifying loaded RISC code...
>Jan 25 13:47:23 hayes-03 kernel: qla2xxx 0000:06:00.0: Allocated (64 KB) for EFT...
>Jan 25 13:47:23 hayes-03 kernel: qla2xxx 0000:06:00.0: Allocated (1413 KB) for firmware dump...
>Jan 25 13:47:23 hayes-03 kernel: scsi5 : qla2xxx
>Jan 25 13:47:23 hayes-03 kernel: qla2xxx 0000:06:00.0: 
>Jan 25 13:47:23 hayes-03 kernel:  QLogic Fibre Channel HBA Driver: 8.02.00-k5-rhel5.2-01
>Jan 25 13:47:23 hayes-03 kernel:   QLogic QLE2460 - PCI-Express to 4Gb FC, Single Channel
>Jan 25 13:47:23 hayes-03 kernel:   ISP2432: PCIe (2.5Gb/s x4) @ 0000:06:00.0 hdma+, host#=5, fw=4.02.02 [Multi-ID] 
>Jan 25 13:48:03 hayes-03 kernel: aoe: 1.1 isbusy
>Jan 25 13:49:03 hayes-03 kernel: aoe: 1.1 isbusy
>Jan 25 13:51:03 hayes-03 last message repeated 2 times
>Jan 25 13:52:03 hayes-03 kernel: aoe: 1.1 isbusy
>Jan 25 13:52:07 hayes-03 clogd[7561]: Starting clogd: 
>Jan 25 13:52:07 hayes-03 clogd[7561]:  Built: Jan 23 2008 16:42:37  
>Jan 25 13:52:12 hayes-03 kernel: dlm: Using TCP for communications
>Jan 25 13:52:12 hayes-03 kernel: dlm: connecting to 2
>Jan 25 13:52:12 hayes-03 kernel: dlm: got connection from 2
>Jan 25 13:52:12 hayes-03 kernel: dlm: connecting to 1
>Jan 25 13:52:13 hayes-03 clvmd: Cluster LVM daemon started - connected to CMAN
>Jan 25 13:52:14 hayes-03 kernel: aoe: 1.1 isbusy
>Jan 25 13:52:16 hayes-03 last message repeated 18 times
>Jan 25 13:52:16 hayes-03 clogd[7561]: Cluster log created (LVM-RJsCglyanNieK87vgK6FBwMG5uS3uWtclZoAeSPdAApPyxI8vf2mTtHMVatVrpvH) 
>Jan 25 13:52:16 hayes-03 clogd[7561]: ****** CPG config callback **************** 
>Jan 25 13:52:16 hayes-03 clogd[7561]: * JOINING (1): 
>Jan 25 13:52:16 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 13:52:16 hayes-03 clogd[7561]: * MEMBERS (2): 
>Jan 25 13:52:16 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 13:52:16 hayes-03 clogd[7561]: *   nodeid: 2, pid: 7472 
>Jan 25 13:52:16 hayes-03 clogd[7561]: * LEAVING (0): 
>Jan 25 13:52:16 hayes-03 clogd[7561]: ***************************************** 
>Jan 25 13:52:16 hayes-03 clogd[7561]: Setting my cluster id: 3 
>Jan 25 13:52:16 hayes-03 dmeventd[7650]: dmeventd ready for processing.
>Jan 25 13:52:16 hayes-03 dmeventd[7650]: Monitoring mirror device hayes-mirror for events 
>Jan 25 13:52:16 hayes-03 lvm[7650]: Loading config file: /etc/lvm/lvm.conf
>Jan 25 13:52:16 hayes-03 lvm[7650]: Opened /etc/lvm/lvm.conf RO
>Jan 25 13:52:16 hayes-03 lvm[7650]: Closed /etc/lvm/lvm.conf
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting log/syslog to 1
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting log/level to 0
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting log/verbose to 0
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting log/indent to 1
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting log/prefix to   
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting log/command_names to 0
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting global/test to 0
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting log/overwrite to 0
>Jan 25 13:52:16 hayes-03 lvm[7650]: log/activation not found in config: defaulting to 0
>Jan 25 13:52:16 hayes-03 lvm[7650]: Logging initialised at Fri Jan 25 13:52:16 2008
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting global/umask to 63
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting devices/dir to /dev
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting global/proc to /proc
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting global/activation to 1
>Jan 25 13:52:16 hayes-03 lvm[7650]: global/suffix not found in config: defaulting to 1
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting global/units to h
>Jan 25 13:52:16 hayes-03 lvm[7650]: activation/readahead not found in config: defaulting to auto
>Jan 25 13:52:16 hayes-03 lvm[7650]: devices/preferred_names not found in config file: using built-in preferences
>Jan 25 13:52:16 hayes-03 lvm[7650]: Matcher built with 3 dfa states
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting devices/ignore_suspended_devices to 0
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting devices/cache_dir to /etc/lvm/cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting devices/write_cache_state to 1
>Jan 25 13:52:16 hayes-03 lvm[7650]: Opened /etc/lvm/cache/.cache RO
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram11: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/etherd/e1.1p5: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/VolGroup00/LogVol01: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR99251-part1: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram10: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/disk/by-path/pci-0000:03:0e.0-scsi-0:0:0:0-part1: Aliased to /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR99251-part1 in device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/disk/by-uuid/d25010ed-d3cc-4b3c-9671-f0b93289ceb6: Aliased to /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR99251-part1 in device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/etherd/e1.1p6: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/VolGroup00/LogVol00: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram12: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram6: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR99251-part2: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram13: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram14: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/disk/by-path/pci-0000:03:0e.0-scsi-0:0:0:0-part2: Aliased to /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR99251-part2 in device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram5: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/mapper/VolGroup00-LogVol00: Aliased to /dev/VolGroup00/LogVol00 in device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/etherd/e1.1p3: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/etherd/e1.1p4: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/etherd/e1.1p7: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram1: Aliased to /dev/ram in device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/sda1: Aliased to /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR99251-part1 in device cache (preferred name)
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ramdisk: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/mapper/VolGroup00-LogVol01: Aliased to /dev/VolGroup00/LogVol01 in device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/etherd/e1.1p2: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram0: Aliased to /dev/ramdisk in device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/root: Aliased to /dev/VolGroup00/LogVol00 in device cache (preferred name)
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/disk/by-label/boot: Aliased to /dev/sda1 in device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram2: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/sda2: Aliased to /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR99251-part2 in device cache (preferred name)
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram8: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram9: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram15: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/etherd/e1.1p1: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram3: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram4: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: /dev/ram7: Added to device cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: Loaded persistent filter cache from /etc/lvm/cache/.cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: Closed /etc/lvm/cache/.cache
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting activation/reserved_stack to 256
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting activation/reserved_memory to 8192
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting activation/process_priority to -18
>Jan 25 13:52:16 hayes-03 lvm[7650]: Initialised format: lvm1
>Jan 25 13:52:16 hayes-03 lvm[7650]: Initialised format: pool
>Jan 25 13:52:16 hayes-03 lvm[7650]: Initialised format: lvm2
>Jan 25 13:52:16 hayes-03 lvm[7650]: global/format not found in config: defaulting to lvm2
>Jan 25 13:52:16 hayes-03 lvm[7650]: Initialised segtype: striped
>Jan 25 13:52:16 hayes-03 lvm[7650]: Initialised segtype: zero
>Jan 25 13:52:16 hayes-03 lvm[7650]: Initialised segtype: error
>Jan 25 13:52:16 hayes-03 lvm[7650]: Initialised segtype: snapshot
>Jan 25 13:52:16 hayes-03 lvm[7650]: Initialised segtype: mirror
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting backup/retain_days to 30
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting backup/retain_min to 10
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting backup/archive_dir to /etc/lvm/archive
>Jan 25 13:52:16 hayes-03 lvm[7650]: Setting backup/backup_dir to /etc/lvm/backup
>Jan 25 13:52:16 hayes-03 lvm[7650]: Locking memory
>Jan 25 13:52:16 hayes-03 lvm[7650]: memlock_count inc to 1
>Jan 25 13:52:16 hayes-03 lvm[7650]: Parsing: _memlock_inc
>Jan 25 13:52:16 hayes-03 lvm[7650]: hayes-mirror is now in-sync 
>Jan 25 13:52:17 hayes-03 clogd[7561]: ****** CPG config callback **************** 
>Jan 25 13:52:17 hayes-03 clogd[7561]: * JOINING (1): 
>Jan 25 13:52:17 hayes-03 clogd[7561]: *   nodeid: 1, pid: 7471 
>Jan 25 13:52:17 hayes-03 clogd[7561]: * MEMBERS (3): 
>Jan 25 13:52:17 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 13:52:17 hayes-03 clogd[7561]: *   nodeid: 2, pid: 7472 
>Jan 25 13:52:17 hayes-03 clogd[7561]: *   nodeid: 1, pid: 7471 
>Jan 25 13:52:17 hayes-03 clogd[7561]: * LEAVING (0): 
>Jan 25 13:52:17 hayes-03 clogd[7561]: ***************************************** 
>Jan 25 13:52:17 hayes-03 clogd[7561]: Checkpoint data available for node 1 
>Jan 25 13:52:17 hayes-03 clogd[7561]: export_checkpoint: sync checkpoint section already exists 
>Jan 25 14:04:27 hayes-03 kernel: GFS 0.1.22-1.el5 (built Jan 24 2008 17:52:33) installed
>Jan 25 14:04:27 hayes-03 kernel: Trying to join cluster "lock_dlm", "HAYES:gfs"
>Jan 25 14:04:27 hayes-03 kernel: Joined cluster. Now mounting FS...
>Jan 25 14:04:28 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: jid=1: Trying to acquire journal lock...
>Jan 25 14:04:28 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: jid=1: Looking at journal...
>Jan 25 14:04:28 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: jid=1: Done
>Jan 25 14:04:28 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: Scanning for log elements...
>Jan 25 14:04:30 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: Found 0 unlinked inodes
>Jan 25 14:04:30 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: Found quota changes for 0 IDs
>Jan 25 14:04:30 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: Done
>Jan 25 14:07:02 hayes-03 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:07:02 hayes-03 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:07:02 hayes-03 kernel: device-mapper: dm-log-clustered: [4478][0][0][0][0][0] [1]
>Jan 25 14:07:06 hayes-03 openais[3790]: [TOTEM] The token was lost in the OPERATIONAL state. 
>Jan 25 14:07:06 hayes-03 openais[3790]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
>Jan 25 14:07:06 hayes-03 openais[3790]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jan 25 14:07:06 hayes-03 openais[3790]: [TOTEM] entering GATHER state from 2. 
>Jan 25 14:07:07 hayes-03 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:07:07 hayes-03 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:07:07 hayes-03 kernel: device-mapper: dm-log-clustered: [4478][0][0][0][0][0] [2]
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] entering GATHER state from 11. 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] Saving state aru 3611 high seq received 3611 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] Storing new sequence id for ring 44c 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] entering COMMIT state. 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] entering RECOVERY state. 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] position [0] member 10.15.89.135: 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] previous ring seq 1096 rep 10.15.89.135 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] aru 3611 high delivered 3611 received flag 1 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] position [1] member 10.15.89.137: 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] previous ring seq 1096 rep 10.15.89.135 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] aru 3611 high delivered 3611 received flag 1 
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] Did not need to originate any messages in recovery. 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:07:10 hayes-03 kernel: dlm: closing connection to node 2
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 14:07:10 hayes-03 openais[3790]: [SYNC ] This node is within the primary component and will provide service. 
>Jan 25 14:07:10 hayes-03 fenced[3818]: fencing deferred to hayes-01
>Jan 25 14:07:10 hayes-03 openais[3790]: [TOTEM] entering OPERATIONAL state. 
>Jan 25 14:07:10 hayes-03 clogd[7561]: ****** CPG config callback **************** 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.135 
>Jan 25 14:07:10 hayes-03 clogd[7561]: * JOINING (0): 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.137 
>Jan 25 14:07:10 hayes-03 clogd[7561]: * MEMBERS (2): 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 3 
>Jan 25 14:07:10 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 14:07:10 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 1 
>Jan 25 14:07:10 hayes-03 clogd[7561]: *   nodeid: 1, pid: 7471 
>Jan 25 14:07:10 hayes-03 clogd[7561]: * LEAVING (1): 
>Jan 25 14:07:10 hayes-03 clogd[7561]: *   nodeid: 2, pid: 7472 
>Jan 25 14:07:10 hayes-03 clogd[7561]: ***************************************** 
>Jan 25 14:08:03 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: jid=0: Trying to acquire journal lock...
>Jan 25 14:08:03 hayes-03 kernel: GFS: fsid=HAYES:gfs.1: jid=0: Busy
>Jan 25 14:09:29 hayes-03 clogd[7561]: Cluster log created (LVM-RJsCglyanNieK87vgK6FBwMG5uS3uWtc0z0RuuaOOrS2qhvxthDYRE5XDI1ch4U9) 
>Jan 25 14:09:29 hayes-03 clogd[7561]: Unable to send request to cluster: Invalid exchange 
>Jan 25 14:09:29 hayes-03 clogd[7561]: Bad callback on local/4 
>Jan 25 14:09:29 hayes-03 clogd[7561]: ****** CPG config callback **************** 
>Jan 25 14:09:29 hayes-03 clogd[7561]: * JOINING (1): 
>Jan 25 14:09:29 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 14:09:29 hayes-03 clogd[7561]: * MEMBERS (1): 
>Jan 25 14:09:29 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 14:09:29 hayes-03 clogd[7561]: * LEAVING (0): 
>Jan 25 14:09:29 hayes-03 clogd[7561]: ***************************************** 
>Jan 25 14:09:29 hayes-03 clogd[7561]: Master resume: reading disk log 
>Jan 25 14:09:29 hayes-03 lvm[7650]: Monitoring mirror device hayes-test for events 
>Jan 25 14:09:29 hayes-03 clogd[7561]: ****** CPG config callback **************** 
>Jan 25 14:09:29 hayes-03 clogd[7561]: * JOINING (1): 
>Jan 25 14:09:29 hayes-03 clogd[7561]: *   nodeid: 1, pid: 7471 
>Jan 25 14:09:29 hayes-03 clogd[7561]: * MEMBERS (2): 
>Jan 25 14:09:29 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 14:09:29 hayes-03 clogd[7561]: *   nodeid: 1, pid: 7471 
>Jan 25 14:09:29 hayes-03 clogd[7561]: * LEAVING (0): 
>Jan 25 14:09:29 hayes-03 clogd[7561]: ***************************************** 
>Jan 25 14:09:29 hayes-03 clogd[7561]: Checkpoint data available for node 1 
>Jan 25 14:09:34 hayes-03 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_SYNC_COUNT]
>Jan 25 14:09:34 hayes-03 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_SYNC_COUNT]
>Jan 25 14:09:34 hayes-03 kernel: device-mapper: dm-log-clustered: [7335][0][0][1][0][0] [3]
>Jan 25 14:09:40 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 5950 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 502 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 129 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 392 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 688 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 295 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 479 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 3 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 1558 
>Jan 25 14:09:40 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:10:09 hayes-03 last message repeated 2 times
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 7780 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 2091 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 146 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 1965 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 2294 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 1853 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 2038 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 3 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 1826 
>Jan 25 14:10:09 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:10:39 hayes-03 last message repeated 2 times
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 9445 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 3760 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 163 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 3617 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 3976 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 3492 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 3683 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 9 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 1851 
>Jan 25 14:10:39 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:11:08 hayes-03 last message repeated 2 times
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 11275 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 5348 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 179 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 5191 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 5584 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 5050 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 5240 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 9 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 2120 
>Jan 25 14:11:08 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:11:38 hayes-03 last message repeated 2 times
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 12943 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 7017 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 195 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 6845 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 7271 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 6690 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 6880 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 9 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 2146 
>Jan 25 14:11:38 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:12:07 hayes-03 last message repeated 2 times
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 14773 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 8606 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 210 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 8420 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 8875 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 8251 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 8441 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 9 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 2411 
>Jan 25 14:12:07 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:12:37 hayes-03 last message repeated 2 times
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 16434 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 10269 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 225 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 10069 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 10553 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 9888 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 10096 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 27 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 2435 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:12:37 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:12:44 hayes-03 lvm[7650]: hayes-test is now in-sync 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] entering GATHER state from 11. 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] Saving state aru 33a29 high seq received 33a29 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] Storing new sequence id for ring 450 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] entering COMMIT state. 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] entering RECOVERY state. 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] position [0] member 10.15.89.135: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] previous ring seq 1100 rep 10.15.89.135 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] aru 33a29 high delivered 33a29 received flag 1 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] position [1] member 10.15.89.136: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] previous ring seq 1100 rep 10.15.89.136 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] aru 9 high delivered 9 received flag 1 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] position [2] member 10.15.89.137: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] previous ring seq 1100 rep 10.15.89.135 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] aru 33a29 high delivered 33a29 received flag 1 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] Did not need to originate any messages in recovery. 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 14:13:31 hayes-03 openais[3790]: [SYNC ] This node is within the primary component and will provide service. 
>Jan 25 14:13:31 hayes-03 openais[3790]: [TOTEM] entering OPERATIONAL state. 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.135 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.136 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.137 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 1 
>Jan 25 14:13:31 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 3 
>Jan 25 14:16:18 hayes-03 kernel: dlm: connecting to 2
>Jan 25 14:16:20 hayes-03 clogd[7561]: ****** CPG config callback **************** 
>Jan 25 14:16:20 hayes-03 clogd[7561]: * JOINING (1): 
>Jan 25 14:16:20 hayes-03 clogd[7561]: *   nodeid: 2, pid: 3779 
>Jan 25 14:16:20 hayes-03 clogd[7561]: * MEMBERS (3): 
>Jan 25 14:16:20 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 14:16:20 hayes-03 clogd[7561]: *   nodeid: 1, pid: 7471 
>Jan 25 14:16:20 hayes-03 clogd[7561]: *   nodeid: 2, pid: 3779 
>Jan 25 14:16:20 hayes-03 clogd[7561]: * LEAVING (0): 
>Jan 25 14:16:20 hayes-03 clogd[7561]: ***************************************** 
>Jan 25 14:16:20 hayes-03 clogd[7561]: Checkpoint data available for node 2 
>Jan 25 14:16:20 hayes-03 clogd[7561]: export_checkpoint: sync checkpoint section already exists 
>Jan 25 14:17:29 hayes-03 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:17:29 hayes-03 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:17:29 hayes-03 kernel: device-mapper: dm-log-clustered: [77645][0][0][1][0][0] [4]
>Jan 25 14:17:34 hayes-03 openais[3790]: [TOTEM] The token was lost in the OPERATIONAL state. 
>Jan 25 14:17:34 hayes-03 openais[3790]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
>Jan 25 14:17:34 hayes-03 openais[3790]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jan 25 14:17:34 hayes-03 openais[3790]: [TOTEM] entering GATHER state from 2. 
>Jan 25 14:17:34 hayes-03 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:17:34 hayes-03 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:17:34 hayes-03 kernel: device-mapper: dm-log-clustered: [77645][0][0][1][0][0] [5]
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] entering GATHER state from 11. 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] Saving state aru d0c1 high seq received d0c1 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] Storing new sequence id for ring 454 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] entering COMMIT state. 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] entering RECOVERY state. 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] position [0] member 10.15.89.135: 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] previous ring seq 1104 rep 10.15.89.135 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] aru d0c1 high delivered d0c1 received flag 1 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] position [1] member 10.15.89.137: 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] previous ring seq 1104 rep 10.15.89.135 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] aru d0c1 high delivered d0c1 received flag 1 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] Did not need to originate any messages in recovery. 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 14:17:39 hayes-03 kernel: dlm: closing connection to node 2
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:17:39 hayes-03 clogd[7561]: ****** CPG config callback **************** 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:17:39 hayes-03 clogd[7561]: * JOINING (0): 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 14:17:39 hayes-03 clogd[7561]: * MEMBERS (2): 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 14:17:39 hayes-03 clogd[7561]: *   nodeid: 3, pid: 7561 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 14:17:39 hayes-03 clogd[7561]: *   nodeid: 1, pid: 7471 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:17:39 hayes-03 clogd[7561]: * LEAVING (1): 
>Jan 25 14:17:39 hayes-03 fenced[3818]: fencing deferred to hayes-01
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] New Configuration: 
>Jan 25 14:17:39 hayes-03 clogd[7561]: *   nodeid: 2, pid: 3779 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:17:39 hayes-03 clogd[7561]: ***************************************** 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] Members Left: 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] Members Joined: 
>Jan 25 14:17:39 hayes-03 openais[3790]: [SYNC ] This node is within the primary component and will provide service. 
>Jan 25 14:17:39 hayes-03 openais[3790]: [TOTEM] entering OPERATIONAL state. 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.135 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CLM  ] got nodejoin message 10.15.89.137 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 3 
>Jan 25 14:17:39 hayes-03 openais[3790]: [CPG  ] got joinlist message from node 1 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 19928 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 11135 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 539 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 10646 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 11535 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 10240 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 10450 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 28 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 5495 
>Jan 25 14:20:19 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:30:52 hayes-03 last message repeated 2 times
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 23797 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 11860 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 978 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 10960 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 12386 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 10240 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 10450 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 28 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 9297 
>Jan 25 14:30:52 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:41:04 hayes-03 last message repeated 2 times
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 27712 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 12555 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 1407 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 11267 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 13203 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 10240 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 10456 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 34 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 13122 
>Jan 25 14:41:04 hayes-03 clogd[7561]:   (null): 0 
>Jan 25 14:50:41 hayes-03 last message repeated 2 times
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_CTR: 2 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_DTR: 0 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_RESUME: 2 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_IN_SYNC: 31663 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_FLUSH: 13230 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_MARK_REGION: 1816 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_CLEAR_REGION: 11561 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_GET_RESYNC_WORK: 13993 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_SET_REGION_SYNC: 10240 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_GET_SYNC_COUNT: 10456 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_STATUS_INFO: 34 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   DM_CLOG_IS_REMOTE_RECOVERING: 17003 
>Jan 25 14:50:41 hayes-03 clogd[7561]:   (null): 0

Actions: View

Attachments on bug 430296: 292998 | 292999 | 293000