Attachment 294665 Details for Bug 432109 – log from taft-02

log from taft-02

taft-02.log (text/plain), 14.10 KB, created by Corey Marthaler on 2008-02-12 15:27:10 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2008-02-12 15:27:10 UTC

Size: 14.10 KB

patch

obsolete

>Feb 11 21:16:20 taft-02 qarshd[14599]: Running cmdline: echo offline > /sys/block/sde/device/state 
>Feb 11 21:16:20 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14599 duration=0(sec)
>Feb 11 21:16:23 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:23 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:25 taft-02 xinetd[6241]: START: qarsh pid=14601 from=10.15.80.47
>Feb 11 21:16:25 taft-02 qarshd[14601]: Talking to peer 10.15.80.47:40487
>Feb 11 21:16:25 taft-02 qarshd[14601]: Running cmdline: dd if=/dev/zero of=/mnt/syncd_log_2legs_1/ddfile count=10 bs=4M 
>Feb 11 21:16:26 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14601 duration=1(sec)
>Feb 11 21:16:26 taft-02 xinetd[6241]: START: qarsh pid=14603 from=10.15.80.47
>Feb 11 21:16:26 taft-02 qarshd[14603]: Talking to peer 10.15.80.47:40488
>Feb 11 21:16:26 taft-02 qarshd[14603]: Running cmdline: sync 
>Feb 11 21:16:26 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:26 taft-02 clogd[6751]: cpg_mcast_joined error: 9 
>Feb 11 21:16:26 taft-02 clogd[6751]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:16:26 taft-02 clogd[6751]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:16:26 taft-02 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:16:26 taft-02 clogd[6751]: Bad callback on local/4 
>Feb 11 21:16:26 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:27 taft-02 last message repeated 2 times
>Feb 11 21:16:27 taft-02 kernel: sd 1:0:0:4: rejecting I/O to offline device
>Feb 11 21:16:27 taft-02 lvm[6831]: No longer monitoring mirror device helter_skelter-syncd_log_2legs_1 for events 
>Feb 11 21:16:41 taft-02 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IN_SYNC:2023650 - retrying
>Feb 11 21:16:41 taft-02 kernel: sd 1:0:0:4: rejecting I/O to offline device
>Feb 11 21:16:41 taft-02 clogd[6751]: rw_log:  write failure: Input/output error 
>Feb 11 21:16:41 taft-02 clogd[6751]: Error writing to disk log 
>Feb 11 21:16:41 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:42 taft-02 kernel: sd 1:0:0:4: rejecting I/O to offline device
>Feb 11 21:16:42 taft-02 clogd[6751]: rw_log:  write failure: Input/output error 
>Feb 11 21:16:42 taft-02 clogd[6751]: Error writing to disk log 
>Feb 11 21:16:42 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:42 taft-02 kernel: sd 1:0:0:4: rejecting I/O to offline device
>Feb 11 21:16:42 taft-02 clogd[6751]: rw_log:  write failure: Input/output error 
>Feb 11 21:16:42 taft-02 clogd[6751]: Error writing to disk log 
>Feb 11 21:16:42 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:42 taft-02 kernel: sd 1:0:0:4: rejecting I/O to offline device
>Feb 11 21:16:42 taft-02 clogd[6751]: rw_log:  write failure: Input/output error 
>Feb 11 21:16:42 taft-02 clogd[6751]: Error writing to disk log 
>Feb 11 21:16:42 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:42 taft-02 clogd[6751]: [jiKwezRI] No match for cluster response: DM_CLOG_GET_RESYNC_WORK:2065539 
>Feb 11 21:16:42 taft-02 clogd[6751]: Current list: 
>Feb 11 21:16:42 taft-02 clogd[6751]:    [none] 
>Feb 11 21:16:42 taft-02 clogd[6751]: [jiKwezRI] Error while processing CPG message, DM_CLOG_GET_RESYNC_WORK: Invalid argument 
>Feb 11 21:16:42 taft-02 clogd[6751]: [jiKwezRI]    Response  : YES 
>Feb 11 21:16:42 taft-02 clogd[6751]: [jiKwezRI]    Originator: 2 
>Feb 11 21:16:42 taft-02 clogd[6751]: [jiKwezRI]    Responder : 1 
>Feb 11 21:16:42 taft-02 clogd[6751]: HISTORY:: 
>Feb 11 21:16:42 taft-02 clogd[6751]: 0:12) SEQ#=2065525, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 1:13) SEQ#=2065526, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 2:14) SEQ#=2065526, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 3:15) SEQ#=2065527, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 4:16) SEQ#=2065527, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 5:17) SEQ#=2065528, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 6:18) SEQ#=2065528, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 7:19) SEQ#=2065529, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 8:0) SEQ#=2065529, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 9:1) SEQ#=2065530, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 10:2) SEQ#=2065530, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 11:3) SEQ#=2065531, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 12:4) SEQ#=2065531, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 13:5) SEQ#=2065532, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 14:6) SEQ#=2065532, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 15:7) SEQ#=2065533, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 16:8) SEQ#=2065533, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 17:9) SEQ#=2065534, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:42 taft-02 clogd[6751]: 18:10) SEQ#=2065534, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:42 taft-02 clogd[6751]: 19:11) SEQ#=2065539, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:16:43 taft-02 lvm[6831]: Monitoring mirror device helter_skelter-syncd_log_2legs_1 for events 
>Feb 11 21:16:57 taft-02 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_GET_RESYNC_WORK:2065539 - retrying
>Feb 11 21:16:58 taft-02 lvm[6831]: No longer monitoring mirror device helter_skelter-syncd_log_2legs_2 for events 
>Feb 11 21:16:59 taft-02 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:59 taft-02 clogd[6751]: [TyB6n0g5] No match for cluster response: DM_CLOG_IS_REMOTE_RECOVERING:2071191 
>Feb 11 21:16:59 taft-02 clogd[6751]: Current list: 
>Feb 11 21:16:59 taft-02 clogd[6751]:    [none] 
>Feb 11 21:16:59 taft-02 clogd[6751]: [TyB6n0g5] Error while processing CPG message, DM_CLOG_IS_REMOTE_RECOVERING: Invalid argument 
>Feb 11 21:16:59 taft-02 clogd[6751]: [TyB6n0g5]    Response  : YES 
>Feb 11 21:16:59 taft-02 clogd[6751]: [TyB6n0g5]    Originator: 2 
>Feb 11 21:16:59 taft-02 clogd[6751]: [TyB6n0g5]    Responder : 1 
>Feb 11 21:16:59 taft-02 clogd[6751]: HISTORY:: 
>Feb 11 21:16:59 taft-02 clogd[6751]: 0:17) SEQ#=2184854, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=1, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 1:18) SEQ#=2184855, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=1, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 2:19) SEQ#=2071187, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 3:0) SEQ#=2406796, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=4, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 4:1) SEQ#=2071187, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:59 taft-02 clogd[6751]: 5:2) SEQ#=2406797, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=4, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 6:3) SEQ#=2184857, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=1, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 7:4) SEQ#=2184858, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=1, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 8:5) SEQ#=2071188, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 9:6) SEQ#=2406798, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=4, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 10:7) SEQ#=2071188, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:59 taft-02 clogd[6751]: 11:8) SEQ#=2406799, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=4, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 12:9) SEQ#=2184860, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=1, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 13:10) SEQ#=2184861, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=1, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 14:11) SEQ#=2071189, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 15:12) SEQ#=2406800, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=4, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 16:13) SEQ#=2071189, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=YES, RSPR=1 
>Feb 11 21:16:59 taft-02 clogd[6751]: 17:14) SEQ#=2406801, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=4, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 18:15) SEQ#=2184864, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=1, RESP=NO 
>Feb 11 21:16:59 taft-02 clogd[6751]: 19:16) SEQ#=2071191, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=NO 
>Feb 11 21:17:13 taft-02 lvm[6831]: Monitoring mirror device helter_skelter-syncd_log_2legs_2 for events 
>Feb 11 21:17:14 taft-02 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:2071191 - retrying
>Feb 11 21:18:04 taft-02 clogd[6751]: cpg_mcast_joined error: 9 
>Feb 11 21:18:04 taft-02 clogd[6751]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:18:04 taft-02 clogd[6751]: [vXFhU9R0] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:18:04 taft-02 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:18:04 taft-02 clogd[6751]: Bad callback on local/4 
>Feb 11 21:18:19 taft-02 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:2097461 - retrying
>Feb 11 21:18:43 taft-02 clogd[6751]: cpg_mcast_joined error: 9 
>Feb 11 21:18:43 taft-02 clogd[6751]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:18:43 taft-02 clogd[6751]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:18:43 taft-02 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:18:43 taft-02 clogd[6751]: Bad callback on local/4 
>Feb 11 21:18:58 taft-02 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IN_SYNC:2113297 - retrying
>Feb 11 21:21:05 taft-02 lvm[6831]: helter_skelter-syncd_log_2legs_2 is now in-sync 
>Feb 11 21:21:05 taft-02 clogd[6751]: cpg_mcast_joined error: 9 
>Feb 11 21:21:05 taft-02 clogd[6751]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:21:05 taft-02 clogd[6751]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:21:05 taft-02 clogd[6751]: Bad callback on local/4 
>Feb 11 21:21:05 taft-02 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:21:05 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14603 duration=279(sec)
>Feb 11 21:21:11 taft-02 xinetd[6241]: START: qarsh pid=14703 from=10.15.80.47
>Feb 11 21:21:11 taft-02 qarshd[14703]: Talking to peer 10.15.80.47:51945
>Feb 11 21:21:11 taft-02 qarshd[14703]: Running cmdline: lvs 
>Feb 11 21:21:11 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14703 duration=0(sec)
>Feb 11 21:21:11 taft-02 xinetd[6241]: START: qarsh pid=14706 from=10.15.80.47
>Feb 11 21:21:11 taft-02 qarshd[14706]: Talking to peer 10.15.80.47:51946
>Feb 11 21:21:11 taft-02 qarshd[14706]: Running cmdline: lvs -a -o +devices 
>Feb 11 21:21:11 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14706 duration=0(sec)
>Feb 11 21:21:11 taft-02 xinetd[6241]: START: qarsh pid=14709 from=10.15.80.47
>Feb 11 21:21:11 taft-02 qarshd[14709]: Talking to peer 10.15.80.47:51947
>Feb 11 21:21:11 taft-02 qarshd[14709]: Running cmdline: lvs -a -o +devices 
>Feb 11 21:21:12 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14709 duration=1(sec)
>Feb 11 21:21:12 taft-02 xinetd[6241]: START: qarsh pid=14712 from=10.15.80.47
>Feb 11 21:21:12 taft-02 qarshd[14712]: Talking to peer 10.15.80.47:51948
>Feb 11 21:21:12 taft-02 qarshd[14712]: Running cmdline: lvs -a -o +devices 
>Feb 11 21:21:12 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14712 duration=0(sec)
>Feb 11 21:21:20 taft-02 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_CLEAR_REGION:2169714 - retrying
>Feb 11 21:22:13 taft-02 qarshd[14594]: Sending child 14595 signal 2
>Feb 11 21:22:13 taft-02 qarshd[14589]: Sending child 14590 signal 2
>Feb 11 21:22:13 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14594 duration=369(sec)
>Feb 11 21:22:13 taft-02 xinetd[6241]: EXIT: qarsh status=0 pid=14589 duration=369(sec)
>Feb 11 21:22:14 taft-02 kernel: GFS: fsid=TAFT:gfs2.1: jid=0: Trying to acquire journal lock...
>Feb 11 21:22:14 taft-02 kernel: GFS: fsid=TAFT:gfs2.1: jid=0: Busy
>Feb 11 21:22:14 taft-02 gfs_controld[6725]: gfs2 finish: needs recovery jid 0 nodeid 1 status 1
>Feb 11 21:22:45 taft-02 lvm[6831]: helter_skelter-syncd_log_2legs_1 is now in-sync 
>Feb 11 21:22:45 taft-02 clogd[6751]: cpg_mcast_joined error: 9 
>Feb 11 21:22:45 taft-02 clogd[6751]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:22:45 taft-02 clogd[6751]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:22:45 taft-02 clogd[6751]: Bad callback on local/4 
>Feb 11 21:22:45 taft-02 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:23:00 taft-02 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_CLEAR_REGION:2194262 - retrying

Actions: View

Attachments on bug 432109: 294664 | 294665 | 294666 | 294667