Attachment 292998 Details for Bug 430296 – log from hayes-01

log from hayes-01

hayes-01.log (text/plain), 27.32 KB, created by Corey Marthaler on 2008-01-25 21:12:33 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2008-01-25 21:12:33 UTC

Size: 27.32 KB

patch

obsolete

>Jan 25 13:50:52 hayes-01 last message repeated 2 times
>Jan 25 13:51:52 hayes-01 kernel: aoe: 1.1 isbusy
>Jan 25 13:51:56 hayes-01 clogd[7471]: Starting clogd: 
>Jan 25 13:51:56 hayes-01 clogd[7471]:  Built: Jan 23 2008 16:42:37  
>Jan 25 13:52:02 hayes-01 kernel: dlm: Using TCP for communications
>Jan 25 13:52:02 hayes-01 kernel: dlm: connecting to 2
>Jan 25 13:52:02 hayes-01 kernel: dlm: got connection from 3
>Jan 25 13:52:02 hayes-01 kernel: dlm: got connection from 2
>Jan 25 13:52:03 hayes-01 clvmd: Cluster LVM daemon started - connected to CMAN
>Jan 25 13:52:03 hayes-01 kernel: aoe: 1.1 isbusy
>Jan 25 13:52:06 hayes-01 last message repeated 30 times
>Jan 25 13:52:06 hayes-01 clogd[7471]: Cluster log created (LVM-RJsCglyanNieK87vgK6FBwMG5uS3uWtclZoAeSPdAApPyxI8vf2mTtHMVatVrpvH) 
>Jan 25 13:52:06 hayes-01 clogd[7471]: Unable to send request to cluster: Invalid exchange 
>Jan 25 13:52:06 hayes-01 clogd[7471]: Bad callback on local/4 
>Jan 25 13:52:06 hayes-01 clogd[7471]: ****** CPG config callback **************** 
>Jan 25 13:52:06 hayes-01 clogd[7471]: * JOINING (1): 
>Jan 25 13:52:06 hayes-01 clogd[7471]: *   nodeid: 1, pid: 7471 
>Jan 25 13:52:06 hayes-01 clogd[7471]: * MEMBERS (3): 
>Jan 25 13:52:06 hayes-01 clogd[7471]: *   nodeid: 1, pid: 7471 
>Jan 25 13:52:06 hayes-01 clogd[7471]: *   nodeid: 2, pid: 7472 
>Jan 25 13:52:06 hayes-01 clogd[7471]: *   nodeid: 3, pid: 7561 
>Jan 25 13:52:06 hayes-01 clogd[7471]: * LEAVING (0): 
>Jan 25 13:52:06 hayes-01 clogd[7471]: ***************************************** 
>Jan 25 13:52:06 hayes-01 clogd[7471]: Setting my cluster id: 1 
>Jan 25 13:52:06 hayes-01 dmeventd[7551]: dmeventd ready for processing.
>Jan 25 13:52:06 hayes-01 dmeventd[7551]: Monitoring mirror device hayes-mirror for events 
>Jan 25 13:52:06 hayes-01 lvm[7551]: Loading config file: /etc/lvm/lvm.conf
>Jan 25 13:52:06 hayes-01 lvm[7551]: Opened /etc/lvm/lvm.conf RO
>Jan 25 13:52:06 hayes-01 lvm[7551]: Closed /etc/lvm/lvm.conf
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting log/syslog to 1
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting log/level to 0
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting log/verbose to 0
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting log/indent to 1
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting log/prefix to   
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting log/command_names to 0
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting global/test to 0
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting log/overwrite to 0
>Jan 25 13:52:06 hayes-01 lvm[7551]: log/activation not found in config: defaulting to 0
>Jan 25 13:52:06 hayes-01 lvm[7551]: Logging initialised at Fri Jan 25 13:52:06 2008
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting global/umask to 63
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting devices/dir to /dev
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting global/proc to /proc
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting global/activation to 1
>Jan 25 13:52:06 hayes-01 lvm[7551]: global/suffix not found in config: defaulting to 1
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting global/units to h
>Jan 25 13:52:06 hayes-01 lvm[7551]: activation/readahead not found in config: defaulting to auto
>Jan 25 13:52:06 hayes-01 lvm[7551]: devices/preferred_names not found in config file: using built-in preferences
>Jan 25 13:52:06 hayes-01 lvm[7551]: Matcher built with 3 dfa states
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting devices/ignore_suspended_devices to 0
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting devices/cache_dir to /etc/lvm/cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting devices/write_cache_state to 1
>Jan 25 13:52:06 hayes-01 lvm[7551]: Opened /etc/lvm/cache/.cache RO
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram11: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/etherd/e1.1p5: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/VolGroup00/LogVol01: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram10: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/disk/by-path/pci-0000:03:0e.0-scsi-0:0:0:0-part1: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/etherd/e1.1p6: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/VolGroup00/LogVol00: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram12: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram6: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram13: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram14: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/disk/by-path/pci-0000:03:0e.0-scsi-0:0:0:0-part2: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram5: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/mapper/VolGroup00-LogVol00: Aliased to /dev/VolGroup00/LogVol00 in device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR98840-part1: Aliased to /dev/disk/by-path/pci-0000:03:0e.0-scsi-0:0:0:0-part1 in device cache (preferred name)
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/etherd/e1.1p3: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/etherd/e1.1p4: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/etherd/e1.1p7: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram1: Aliased to /dev/ram in device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/sda1: Aliased to /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR98840-part1 in device cache (preferred name)
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ramdisk: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/mapper/VolGroup00-LogVol01: Aliased to /dev/VolGroup00/LogVol01 in device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/disk/by-uuid/31626aa7-d8da-4a9c-b7ee-6b863f87a008: Aliased to /dev/sda1 in device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/etherd/e1.1p2: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram0: Aliased to /dev/ramdisk in device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/root: Aliased to /dev/VolGroup00/LogVol00 in device cache (preferred name)
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR98840-part2: Aliased to /dev/disk/by-path/pci-0000:03:0e.0-scsi-0:0:0:0-part2 in device cache (preferred name)
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/disk/by-label/boot: Aliased to /dev/sda1 in device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram2: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/sda2: Aliased to /dev/disk/by-id/scsi-SATA_WDC_WD800JD-75M_WD-WMAM9VR98840-part2 in device cache (preferred name)
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram8: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram9: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram15: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/etherd/e1.1p1: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram3: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram4: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: /dev/ram7: Added to device cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: Loaded persistent filter cache from /etc/lvm/cache/.cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: Closed /etc/lvm/cache/.cache
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting activation/reserved_stack to 256
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting activation/reserved_memory to 8192
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting activation/process_priority to -18
>Jan 25 13:52:06 hayes-01 lvm[7551]: Initialised format: lvm1
>Jan 25 13:52:06 hayes-01 lvm[7551]: Initialised format: pool
>Jan 25 13:52:06 hayes-01 lvm[7551]: Initialised format: lvm2
>Jan 25 13:52:06 hayes-01 lvm[7551]: global/format not found in config: defaulting to lvm2
>Jan 25 13:52:06 hayes-01 lvm[7551]: Initialised segtype: striped
>Jan 25 13:52:06 hayes-01 lvm[7551]: Initialised segtype: zero
>Jan 25 13:52:06 hayes-01 lvm[7551]: Initialised segtype: error
>Jan 25 13:52:06 hayes-01 lvm[7551]: Initialised segtype: snapshot
>Jan 25 13:52:06 hayes-01 lvm[7551]: Initialised segtype: mirror
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting backup/retain_days to 30
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting backup/retain_min to 10
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting backup/archive_dir to /etc/lvm/archive
>Jan 25 13:52:06 hayes-01 lvm[7551]: Setting backup/backup_dir to /etc/lvm/backup
>Jan 25 13:52:06 hayes-01 lvm[7551]: Locking memory
>Jan 25 13:52:06 hayes-01 lvm[7551]: memlock_count inc to 1
>Jan 25 13:52:06 hayes-01 lvm[7551]: Parsing: _memlock_inc
>Jan 25 13:52:11 hayes-01 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_SYNC_COUNT]
>Jan 25 13:52:11 hayes-01 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_SYNC_COUNT]
>Jan 25 13:52:11 hayes-01 kernel: device-mapper: dm-log-clustered: [4][0][0][0][0][0] [1]
>Jan 25 13:52:11 hayes-01 lvm[7551]: hayes-mirror is now in-sync 
>Jan 25 14:04:16 hayes-01 kernel: GFS 0.1.22-1.el5 (built Jan 24 2008 17:52:33) installed
>Jan 25 14:04:16 hayes-01 kernel: Trying to join cluster "lock_dlm", "HAYES:gfs"
>Jan 25 14:04:16 hayes-01 kernel: Joined cluster. Now mounting FS...
>Jan 25 14:04:17 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=2: Trying to acquire journal lock...
>Jan 25 14:04:17 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=2: Looking at journal...
>Jan 25 14:04:17 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=2: Done
>Jan 25 14:06:53 hayes-01 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:06:53 hayes-01 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:06:53 hayes-01 kernel: device-mapper: dm-log-clustered: [354][0][0][0][0][0] [2]
>Jan 25 14:06:55 hayes-01 openais[3712]: [TOTEM] entering GATHER state from 12. 
>Jan 25 14:06:58 hayes-01 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:06:58 hayes-01 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_RESYNC_WORK]
>Jan 25 14:06:58 hayes-01 kernel: device-mapper: dm-log-clustered: [354][0][0][0][0][0] [3]
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] entering GATHER state from 0. 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] Creating commit token because I am the rep. 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] Saving state aru 3611 high seq received 3611 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] Storing new sequence id for ring 44c 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] entering COMMIT state. 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] entering RECOVERY state. 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] position [0] member 10.15.89.135: 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] previous ring seq 1096 rep 10.15.89.135 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] aru 3611 high delivered 3611 received flag 1 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] position [1] member 10.15.89.137: 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] previous ring seq 1096 rep 10.15.89.135 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] aru 3611 high delivered 3611 received flag 1 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] Did not need to originate any messages in recovery. 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] Sending initial ORF token 
>Jan 25 14:07:00 hayes-01 kernel: dlm: closing connection to node 2
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:07:00 hayes-01 clogd[7471]: cluster_send failed 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] New Configuration: 
>Jan 25 14:07:00 hayes-01 clogd[7471]: Error while processing CPG message 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] Members Left: 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] Members Joined: 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] New Configuration: 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] Members Left: 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] Members Joined: 
>Jan 25 14:07:00 hayes-01 openais[3712]: [SYNC ] This node is within the primary component and will provide service. 
>Jan 25 14:07:00 hayes-01 clogd[7471]: ****** CPG config callback **************** 
>Jan 25 14:07:00 hayes-01 openais[3712]: [TOTEM] entering OPERATIONAL state. 
>Jan 25 14:07:00 hayes-01 clogd[7471]: * JOINING (0): 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] got nodejoin message 10.15.89.135 
>Jan 25 14:07:00 hayes-01 clogd[7471]: * MEMBERS (2): 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CLM  ] got nodejoin message 10.15.89.137 
>Jan 25 14:07:00 hayes-01 clogd[7471]: *   nodeid: 1, pid: 7471 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CPG  ] got joinlist message from node 3 
>Jan 25 14:07:00 hayes-01 clogd[7471]: *   nodeid: 3, pid: 7561 
>Jan 25 14:07:00 hayes-01 openais[3712]: [CPG  ] got joinlist message from node 1 
>Jan 25 14:07:00 hayes-01 clogd[7471]: * LEAVING (1): 
>Jan 25 14:07:00 hayes-01 clogd[7471]: *   nodeid: 2, pid: 7472 
>Jan 25 14:07:00 hayes-01 clogd[7471]: ***************************************** 
>Jan 25 14:07:30 hayes-01 fenced[3745]: hayes-02 not a cluster member after 30 sec post_fail_delay
>Jan 25 14:07:30 hayes-01 fenced[3745]: fencing node "hayes-02"
>Jan 25 14:07:52 hayes-01 fenced[3745]: fence "hayes-02" success
>Jan 25 14:07:52 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=0: Trying to acquire journal lock...
>Jan 25 14:07:53 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=0: Looking at journal...
>Jan 25 14:07:53 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=0: Acquiring the transaction lock...
>Jan 25 14:07:53 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=0: Replaying journal...
>Jan 25 14:07:53 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=0: Replayed 3 of 3 blocks
>Jan 25 14:07:53 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=0: replays = 3, skips = 0, sames = 0
>Jan 25 14:07:53 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=0: Journal replayed in 1s
>Jan 25 14:07:53 hayes-01 kernel: GFS: fsid=HAYES:gfs.2: jid=0: Done
>Jan 25 14:09:19 hayes-01 clogd[7471]: Cluster log created (LVM-RJsCglyanNieK87vgK6FBwMG5uS3uWtc0z0RuuaOOrS2qhvxthDYRE5XDI1ch4U9) 
>Jan 25 14:09:19 hayes-01 clogd[7471]: Unable to send request to cluster: Invalid exchange 
>Jan 25 14:09:19 hayes-01 clogd[7471]: Bad callback on local/4 
>Jan 25 14:09:19 hayes-01 clogd[7471]: ****** CPG config callback **************** 
>Jan 25 14:09:19 hayes-01 clogd[7471]: * JOINING (1): 
>Jan 25 14:09:19 hayes-01 clogd[7471]: *   nodeid: 1, pid: 7471 
>Jan 25 14:09:19 hayes-01 clogd[7471]: * MEMBERS (2): 
>Jan 25 14:09:19 hayes-01 clogd[7471]: *   nodeid: 1, pid: 7471 
>Jan 25 14:09:19 hayes-01 clogd[7471]: *   nodeid: 3, pid: 7561 
>Jan 25 14:09:19 hayes-01 clogd[7471]: * LEAVING (0): 
>Jan 25 14:09:19 hayes-01 clogd[7471]: ***************************************** 
>Jan 25 14:09:19 hayes-01 lvm[7551]: Monitoring mirror device hayes-test for events 
>Jan 25 14:09:24 hayes-01 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_GET_SYNC_COUNT]
>Jan 25 14:09:24 hayes-01 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_GET_SYNC_COUNT]
>Jan 25 14:09:24 hayes-01 kernel: device-mapper: dm-log-clustered: [3385][1][0][0][0][0] [4]
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] entering GATHER state from 11. 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] Creating commit token because I am the rep. 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] Saving state aru 33a29 high seq received 33a29 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] Storing new sequence id for ring 450 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] entering COMMIT state. 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] entering RECOVERY state. 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] position [0] member 10.15.89.135: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] previous ring seq 1100 rep 10.15.89.135 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] aru 33a29 high delivered 33a29 received flag 1 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] position [1] member 10.15.89.136: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] previous ring seq 1100 rep 10.15.89.136 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] aru 9 high delivered 9 received flag 1 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] position [2] member 10.15.89.137: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] previous ring seq 1100 rep 10.15.89.135 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] aru 33a29 high delivered 33a29 received flag 1 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] Did not need to originate any messages in recovery. 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] Sending initial ORF token 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] New Configuration: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] Members Left: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] Members Joined: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] New Configuration: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] Members Left: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] Members Joined: 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 14:13:20 hayes-01 openais[3712]: [SYNC ] This node is within the primary component and will provide service. 
>Jan 25 14:13:20 hayes-01 openais[3712]: [TOTEM] entering OPERATIONAL state. 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] got nodejoin message 10.15.89.135 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] got nodejoin message 10.15.89.136 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CLM  ] got nodejoin message 10.15.89.137 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CPG  ] got joinlist message from node 1 
>Jan 25 14:13:20 hayes-01 openais[3712]: [CPG  ] got joinlist message from node 3 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   (null): 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_CTR: 2 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_DTR: 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_RESUME: 2 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_IN_SYNC: 4041 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_FLUSH: 630 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_MARK_REGION: 381 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_CLEAR_REGION: 276 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_GET_RESYNC_WORK: 838 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_SET_REGION_SYNC: 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_GET_SYNC_COUNT: 14 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_STATUS_INFO: 9 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   DM_CLOG_IS_REMOTE_RECOVERING: 3807 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   (null): 0 
>Jan 25 14:15:27 hayes-01 clogd[7471]:   (null): 0 
>Jan 25 14:16:07 hayes-01 kernel: dlm: connecting to 2
>Jan 25 14:16:07 hayes-01 kernel: dlm: got connection from 2
>Jan 25 14:16:08 hayes-01 lvm[7551]: hayes-test is now in-sync 
>Jan 25 14:16:09 hayes-01 clogd[7471]: ****** CPG config callback **************** 
>Jan 25 14:16:09 hayes-01 clogd[7471]: * JOINING (1): 
>Jan 25 14:16:09 hayes-01 clogd[7471]: *   nodeid: 2, pid: 3779 
>Jan 25 14:16:09 hayes-01 clogd[7471]: * MEMBERS (3): 
>Jan 25 14:16:09 hayes-01 clogd[7471]: *   nodeid: 1, pid: 7471 
>Jan 25 14:16:09 hayes-01 clogd[7471]: *   nodeid: 3, pid: 7561 
>Jan 25 14:16:09 hayes-01 clogd[7471]: *   nodeid: 2, pid: 3779 
>Jan 25 14:16:09 hayes-01 clogd[7471]: * LEAVING (0): 
>Jan 25 14:16:09 hayes-01 clogd[7471]: ***************************************** 
>Jan 25 14:16:09 hayes-01 clogd[7471]: Checkpoint data available for node 2 
>Jan 25 14:17:18 hayes-01 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_IN_SYNC]
>Jan 25 14:17:18 hayes-01 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_IN_SYNC]
>Jan 25 14:17:18 hayes-01 kernel: device-mapper: dm-log-clustered: [11855][1][0][0][0][0] [5]
>Jan 25 14:17:23 hayes-01 openais[3712]: [TOTEM] The token was lost in the OPERATIONAL state. 
>Jan 25 14:17:23 hayes-01 openais[3712]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
>Jan 25 14:17:23 hayes-01 openais[3712]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
>Jan 25 14:17:23 hayes-01 kernel: device-mapper: dm-log-clustered: Timed out waiting for cluster log server [DM_CLOG_IN_SYNC]
>Jan 25 14:17:23 hayes-01 kernel: device-mapper: dm-log-clustered: Retrying request [DM_CLOG_IN_SYNC]
>Jan 25 14:17:23 hayes-01 openais[3712]: [TOTEM] entering GATHER state from 2. 
>Jan 25 14:17:23 hayes-01 kernel: device-mapper: dm-log-clustered: [11855][1][0][0][0][0] [6]
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] entering GATHER state from 0. 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] Creating commit token because I am the rep. 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] Saving state aru d0c1 high seq received d0c1 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] Storing new sequence id for ring 454 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] entering COMMIT state. 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] entering RECOVERY state. 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] position [0] member 10.15.89.135: 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] previous ring seq 1104 rep 10.15.89.135 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] aru d0c1 high delivered d0c1 received flag 1 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] position [1] member 10.15.89.137: 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] previous ring seq 1104 rep 10.15.89.135 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] aru d0c1 high delivered d0c1 received flag 1 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] Did not need to originate any messages in recovery. 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] Sending initial ORF token 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] New Configuration: 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:17:28 hayes-01 kernel: dlm: closing connection to node 2
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:17:28 hayes-01 clogd[7471]: cluster_send failed 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] Members Left: 
>Jan 25 14:17:28 hayes-01 clogd[7471]: Error while processing CPG message 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.136)  
>Jan 25 14:17:28 hayes-01 clogd[7471]: cluster_send failed 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] Members Joined: 
>Jan 25 14:17:28 hayes-01 clogd[7471]: Error while processing CPG message 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] CLM CONFIGURATION CHANGE 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] New Configuration: 
>Jan 25 14:17:28 hayes-01 clogd[7471]: ****** CPG config callback **************** 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.135)  
>Jan 25 14:17:28 hayes-01 clogd[7471]: * JOINING (0): 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] 	r(0) ip(10.15.89.137)  
>Jan 25 14:17:28 hayes-01 clogd[7471]: * MEMBERS (2): 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] Members Left: 
>Jan 25 14:17:28 hayes-01 clogd[7471]: *   nodeid: 1, pid: 7471 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] Members Joined: 
>Jan 25 14:17:28 hayes-01 clogd[7471]: *   nodeid: 3, pid: 7561 
>Jan 25 14:17:28 hayes-01 openais[3712]: [SYNC ] This node is within the primary component and will provide service. 
>Jan 25 14:17:28 hayes-01 clogd[7471]: * LEAVING (1): 
>Jan 25 14:17:28 hayes-01 openais[3712]: [TOTEM] entering OPERATIONAL state. 
>Jan 25 14:17:28 hayes-01 clogd[7471]: *   nodeid: 2, pid: 3779 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] got nodejoin message 10.15.89.135 
>Jan 25 14:17:28 hayes-01 clogd[7471]: ***************************************** 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CLM  ] got nodejoin message 10.15.89.137 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CPG  ] got joinlist message from node 3 
>Jan 25 14:17:28 hayes-01 openais[3712]: [CPG  ] got joinlist message from node 1 
>Jan 25 14:17:58 hayes-01 fenced[3745]: hayes-02 not a cluster member after 30 sec post_fail_delay
>Jan 25 14:17:58 hayes-01 fenced[3745]: fencing node "hayes-02"
>Jan 25 14:18:20 hayes-01 fenced[3745]: fence "hayes-02" success
>Jan 25 14:26:35 hayes-01 clogd[7471]:   (null): 0 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_CTR: 2 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_DTR: 0 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_RESUME: 2 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_IN_SYNC: 7932 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_FLUSH: 1363 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_MARK_REGION: 818 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_CLEAR_REGION: 587 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_GET_RESYNC_WORK: 1724 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_SET_REGION_SYNC: 0 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_GET_SYNC_COUNT: 16 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_STATUS_INFO: 10 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   DM_CLOG_IS_REMOTE_RECOVERING: 7546 
>Jan 25 14:26:35 hayes-01 clogd[7471]:   (null): 0 
>Jan 25 14:35:26 hayes-01 last message repeated 2 times
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_CTR: 2 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_DTR: 0 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_PRESUSPEND: 0 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_POSTSUSPEND: 0 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_RESUME: 2 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_GET_REGION_SIZE: 0 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_IS_CLEAN: 0 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_IN_SYNC: 12024 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_FLUSH: 1963 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_MARK_REGION: 1179 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_CLEAR_REGION: 843 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_GET_RESYNC_WORK: 2430 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_SET_REGION_SYNC: 0 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_GET_SYNC_COUNT: 16 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_STATUS_INFO: 10 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_STATUS_TABLE: 0 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   DM_CLOG_IS_REMOTE_RECOVERING: 11531 
>Jan 25 14:35:26 hayes-01 clogd[7471]:   (null): 0

Actions: View

Attachments on bug 430296: 292998 | 292999 | 293000