Attachment 1487721 Details for Bug 1633583 – cluster

cluster

log (text/plain), 8.98 KB, created by Roman Bednář on 2018-09-27 10:28:27 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Roman Bednář

Created: 2018-09-27 10:28:27 UTC

Size: 8.98 KB

patch

obsolete

>Sep 26 16:30:24 virt-369.cluster-qe.lab.eng.brq.redhat.com multipathd[1708]: dm-2: remove map (uevent)
>Sep 26 16:30:24 virt-369.cluster-qe.lab.eng.brq.redhat.com multipathd[1708]: dm-2: devmap not registered, can't remove
>Sep 26 16:30:24 virt-369.cluster-qe.lab.eng.brq.redhat.com multipathd[1708]: dm-2: remove map (uevent)
>Sep 26 16:30:43 virt-369.cluster-qe.lab.eng.brq.redhat.com corosync[2167]:  [TOTEM ] A processor failed, forming new configuration.
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com kernel: dlm: closing connection to node 1
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com corosync[2167]:  [TOTEM ] A new membership (10.37.166.106:20) was formed. Members left: 1
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com corosync[2167]:  [TOTEM ] Failed to receive the leave message. failed: 1
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com corosync[2167]:  [QUORUM] Members[1]: 2
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com corosync[2167]:  [MAIN  ] Completed service synchronization, ready to provide service.
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com dlm_controld[2744]: 6414 fence request 1 pid 31452 nodedown time 1537972244 fence_all dlm_stonith
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com pacemakerd[2214]:   notice: Node virt-368 state is now lost
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com attrd[2221]:   notice: Node virt-368 state is now lost
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com attrd[2221]:   notice: Removing all virt-368 attributes for peer loss
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com attrd[2221]:   notice: Lost attribute writer virt-368
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com attrd[2221]:   notice: Purged 1 peer with id=1 and/or uname=virt-368 from the membership cache
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Node virt-368 state is now lost
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com cib[2217]:   notice: Node virt-368 state is now lost
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com cib[2217]:   notice: Purged 1 peer with id=1 and/or uname=virt-368 from the membership cache
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Node virt-368 state is now lost
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Purged 1 peer with id=1 and/or uname=virt-368 from the membership cache
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:  warning: No reason to expect node 1 to be down
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Stonith/shutdown of virt-368 not matched
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: State transition S_IDLE -> S_POLICY_ENGINE
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:  warning: No reason to expect node 1 to be down
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Stonith/shutdown of virt-368 not matched
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com dlm_stonith[31452]: stonith_api_time: Found 0 entries for 1/(null): 0 in progress, 0 completed
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Client stonith-api.31452.5b13fa5c wants to fence (reboot) '1' with device '(any)'
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Requesting peer fencing (reboot) of virt-368
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: fence-virt-369 can not fence (reboot) virt-368: static-list
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: fence-virt-368 can fence (reboot) virt-368 (aka. 'virt-368.cluster-qe.lab.eng.brq.redhat.com'): static-list
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: fence-virt-369 can not fence (reboot) virt-368: static-list
>Sep 26 16:30:44 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: fence-virt-368 can fence (reboot) virt-368 (aka. 'virt-368.cluster-qe.lab.eng.brq.redhat.com'): static-list
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Cluster node virt-368 will be fenced: peer is no longer part of the cluster
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Node virt-368 is unclean
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Action fence-virt-368_stop_0 on virt-368 is unrunnable (offline)
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Action dlm:0_stop_0 on virt-368 is unrunnable (offline)
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Action dlm:0_stop_0 on virt-368 is unrunnable (offline)
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Action clvmd:0_stop_0 on virt-368 is unrunnable (offline)
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Action clvmd:0_stop_0 on virt-368 is unrunnable (offline)
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Scheduling Node virt-368 for STONITH
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:   notice:  * Fence (reboot) virt-368 'peer is no longer part of the cluster'
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:   notice:  * Move       fence-virt-368     ( virt-368 -> virt-369 )
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:   notice:  * Stop       dlm:0              (             virt-368 )   due to node availability
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:   notice:  * Stop       clvmd:0            (             virt-368 )   due to node availability
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com pengine[2222]:  warning: Calculated transition 9 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-0.bz2
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Requesting fencing (reboot) of node virt-368
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Client crmd.2223.ddde95e5 wants to fence (reboot) 'virt-368' with device '(any)'
>Sep 26 16:30:45 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Merging stonith action reboot for node virt-368 originating from client crmd.2223.fb7d44bf with identical request from stonith-api.31452@virt-369.d85
>Sep 26 16:30:49 virt-369.cluster-qe.lab.eng.brq.redhat.com root[31503]: 2018-09-26 16:30:49 /usr/bin/rhts-test-runner.sh 2592046 6420 hearbeat...
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Operation 'reboot' [31454] (call 2 from stonith-api.31452) for host 'virt-368' with device 'fence-virt-368' returned: 0 (OK)
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Operation reboot of virt-368 by virt-369 for stonith-api.31452@virt-369.d85389ce: OK
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-ng[2218]:   notice: Operation reboot of virt-368 by virt-369 for crmd.2223@virt-369.fb7d44bf: OK
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com stonith-api[31452]: stonith_api_kick: Node 1/(null) kicked: reboot
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Peer virt-368 was terminated (reboot) by virt-369 on behalf of stonith-api.31452: OK
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Transition aborted: External Fencing Operation
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Stonith operation 2/1:9:0:2c66e24c-1cca-4d8b-80e9-c1c2c2cff7c5: OK (0)
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Peer virt-368 was terminated (reboot) by virt-369 on behalf of crmd.2223: OK
>Sep 26 16:30:52 virt-369.cluster-qe.lab.eng.brq.redhat.com crmd[2223]:   notice: Transition 9 (Complete=10, Pending=0, Fired=0, Skipped=1, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
>Sep 26 16:30:53 virt-369.cluster-qe.lab.eng.brq.redhat.com dlm_controld[2744]: 6423 fence result 1 pid 31452 result 0 exit status
>Sep 26 16:30:53 virt-369.cluster-qe.lab.eng.brq.redhat.com kernel: dlm: clvmd: dlm_recover 3
>Sep 26 16:30:53 virt-369.cluster-qe.lab.eng.brq.redhat.com kernel: dlm: clvmd: dlm_clear_toss 6 done
>Sep 26 16:30:53 virt-369.cluster-qe.lab.eng.brq.redhat.com kernel: dlm: clvmd: remove member 1
>Sep 26 16:30:53 virt-369.cluster-qe.lab.eng.brq.redhat.com kernel: dlm: clvmd: dlm_recover_members 1 nodes
>Sep 26 16:30:53 virt-369.cluster-qe.lab.eng.brq.redhat.com kernel: dlm: clvmd: generation 3 slots 1 2:2
>Sep 26 16:30:53 virt-369.cluster-qe.lab.eng.brq.redhat.com kernel: dlm: clvmd: dlm_recover_directory
>Sep 26 16:30:53 virt-369.cluster-qe.lab.eng.brq.redhat.com kernel: dlm: clvmd: dlm_recover_directory 0 in 0 new

Actions: View

Attachments on bug 1633583: 1487720 | 1487721 | 1487823