Attachment 294666 Details for Bug 432109 – log from taft-03

log from taft-03

taft-03.log (text/plain), 10.04 KB, created by Corey Marthaler on 2008-02-12 15:28:40 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2008-02-12 15:28:40 UTC

Size: 10.04 KB

patch

obsolete

>Feb 11 21:16:40 taft-03 qarshd[15435]: Running cmdline: echo offline > /sys/block/sde/device/state 
>Feb 11 21:16:40 taft-03 xinetd[6229]: EXIT: qarsh status=0 pid=15435 duration=0(sec)
>Feb 11 21:16:44 taft-03 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:44 taft-03 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_FLUSH]: -5
>Feb 11 21:16:47 taft-03 kernel: sd 1:0:0:4: rejecting I/O to offline device
>Feb 11 21:16:47 taft-03 lvm[7060]: No longer monitoring mirror device helter_skelter-syncd_log_2legs_1 for events 
>Feb 11 21:17:17 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_RESUME:3901108 - retrying
>Feb 11 21:17:17 taft-03 lvm[7060]: Monitoring mirror device helter_skelter-syncd_log_2legs_1 for events 
>Feb 11 21:17:18 taft-03 lvm[7060]: No longer monitoring mirror device helter_skelter-syncd_log_2legs_2 for events 
>Feb 11 21:17:18 taft-03 clogd[6738]: [TyB6n0g5] No match for cluster response: DM_CLOG_IS_REMOTE_RECOVERING:3901348 
>Feb 11 21:17:18 taft-03 clogd[6738]: Current list: 
>Feb 11 21:17:18 taft-03 clogd[6738]:    [none] 
>Feb 11 21:17:18 taft-03 clogd[6738]: [TyB6n0g5] Error while processing CPG message, DM_CLOG_IS_REMOTE_RECOVERING: Invalid argument 
>Feb 11 21:17:18 taft-03 clogd[6738]: [TyB6n0g5]    Response  : YES 
>Feb 11 21:17:18 taft-03 clogd[6738]: [TyB6n0g5]    Originator: 3 
>Feb 11 21:17:18 taft-03 clogd[6738]: [TyB6n0g5]    Responder : 1 
>Feb 11 21:17:18 taft-03 clogd[6738]: HISTORY:: 
>Feb 11 21:17:18 taft-03 clogd[6738]: 0:11) SEQ#=3901344, UUID=TyB6n0g5, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=3, RESP=YES, RSPR=1 
>Feb 11 21:17:18 taft-03 clogd[6738]: 1:12) SEQ#=2184614, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=1, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 2:13) SEQ#=2184613, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=1, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 3:14) SEQ#=2071039, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 4:15) SEQ#=2071038, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 5:16) SEQ#=3901346, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=3, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 6:17) SEQ#=3901346, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=3, RESP=YES, RSPR=1 
>Feb 11 21:17:18 taft-03 clogd[6738]: 7:18) SEQ#=2406635, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=4, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 8:19) SEQ#=2406636, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=4, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 9:0) SEQ#=2184616, UUID=jiKwezRI, TYPE=DM_CLOG_SET_REGION_SYNC, ORIG=1, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 10:1) SEQ#=2184617, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=1, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 11:2) SEQ#=2071040, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 12:3) SEQ#=2071041, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 13:4) SEQ#=3901347, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=3, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 14:5) SEQ#=3901347, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=3, RESP=YES, RSPR=1 
>Feb 11 21:17:18 taft-03 clogd[6738]: 15:6) SEQ#=2071042, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=2, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 16:7) SEQ#=2071043, UUID=jiKwezRI, TYPE=DM_CLOG_GET_RESYNC_WORK, ORIG=2, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 17:8) SEQ#=3901348, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=3, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 18:9) SEQ#=2406637, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=4, RESP=NO 
>Feb 11 21:17:18 taft-03 clogd[6738]: 19:10) SEQ#=2184620, UUID=TyB6n0g5, TYPE=DM_CLOG_IS_REMOTE_RECOVERING, ORIG=1, RESP=NO 
>Feb 11 21:17:33 taft-03 lvm[7060]: Monitoring mirror device helter_skelter-syncd_log_2legs_2 for events 
>Feb 11 21:17:33 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:3901348 - retrying
>Feb 11 21:17:33 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:17:33 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:17:33 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:17:33 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:17:33 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:17:48 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IN_SYNC:3901439 - retrying
>Feb 11 21:18:01 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:18:01 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:18:01 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:18:01 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:18:01 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:18:16 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:3914251 - retrying
>Feb 11 21:18:38 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:18:38 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:18:38 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:18:38 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:18:38 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:18:53 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:3930444 - retrying
>Feb 11 21:18:56 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:18:56 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:18:56 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:18:56 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:18:56 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:19:11 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:3935671 - retrying
>Feb 11 21:19:11 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:19:11 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:19:11 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:19:11 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:19:11 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:19:26 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:3941533 - retrying
>Feb 11 21:19:29 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:19:29 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:19:29 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:19:29 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:19:29 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:19:44 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:3951463 - retrying
>Feb 11 21:19:50 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:19:50 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:19:50 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:19:50 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:19:50 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:20:05 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IN_SYNC:3961557 - retrying
>Feb 11 21:20:05 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:20:05 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:20:05 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:20:05 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:20:05 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:20:20 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IN_SYNC:3969038 - retrying
>Feb 11 21:20:21 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:20:21 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:20:21 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:20:21 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:20:21 taft-03 kernel: device-mapper: dm-log-clustered: Stray request returned: <NULL>, 0
>Feb 11 21:20:36 taft-03 kernel: device-mapper: dm-log-clustered: Request timed out on DM_CLOG_IS_REMOTE_RECOVERING:3975114 - retrying
>Feb 11 21:21:27 taft-03 lvm[7060]: helter_skelter-syncd_log_2legs_2 is now in-sync 
>Feb 11 21:21:27 taft-03 clogd[6738]: cpg_mcast_joined error: 9 
>Feb 11 21:21:27 taft-03 clogd[6738]: cluster_send failed at: local.c:212 (do_local_work) 
>Feb 11 21:21:27 taft-03 clogd[6738]: [] Unable to send (null) to cluster: Invalid exchange 
>Feb 11 21:21:27 taft-03 clogd[6738]: Bad callback on local/4 
>Feb 11 21:21:27 taft-03 kernel: device-mapper: dm-log-clustered: Server error while processing request [DM_CLOG_IS_REMOTE_RECOVERING]: -52
>Feb 11 21:22:33 taft-03 qarshd[15428]: Sending child 15429 signal 2
>Feb 11 21:22:33 taft-03 qarshd[15423]: Sending child 15424 signal 2
>Feb 11 21:22:33 taft-03 xinetd[6229]: EXIT: qarsh status=0 pid=15423 duration=369(sec)
>Feb 11 21:22:33 taft-03 xinetd[6229]: EXIT: qarsh status=0 pid=15428 duration=369(sec)
>Feb 11 21:22:34 taft-03 kernel: GFS: fsid=TAFT:gfs2.2: jid=0: Trying to acquire journal lock...
>Feb 11 21:22:34 taft-03 kernel: GFS: fsid=TAFT:gfs2.2: jid=0: Busy
>Feb 11 21:22:34 taft-03 gfs_controld[6711]: gfs2 finish: needs recovery jid 0 nodeid 1 status 1
>Feb 11 21:22:56 taft-03 lvm[7060]: helter_skelter-syncd_log_2legs_1 is now in-sync

Actions: View

Attachments on bug 432109: 294664 | 294665 | 294666 | 294667