Bug 1300688 - A systemd high availability resource is migrated to a standby node where it is not necessarily installed, fails and becomes unmanaged
A systemd high availability resource is migrated to a standby node where it i...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pacemaker (Show other bugs)
7.2
x86_64 Linux
unspecified Severity urgent
: rc
: ---
Assigned To: Ken Gaillot
cluster-qe@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-21 08:13 EST by Matti Linnanvuori
Modified: 2016-07-25 10:05 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-25 10:05:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
crm_report output (686.44 KB, application/x-bzip)
2016-01-26 02:12 EST, Matti Linnanvuori
no flags Details
crm_report output with debug logging (1023.30 KB, application/x-bzip)
2016-01-28 09:09 EST, Matti Linnanvuori
no flags Details

  None (edit)
Description Matti Linnanvuori 2016-01-21 08:13:38 EST
Description of problem:

A systemd high availability resource is migrated to a standby node where it is not necessarily installed, fails and becomes unmanaged.

Version-Release number of selected component (if applicable):

pacemaker 1.1.13 10.el7

How reproducible:

Usually reproducible.

Steps to Reproduce:
1. Create a three-node cluster with a standby node.
2. Create a systemd resource.
3. Watch the resource.

Actual results:

A systemd high availability resource is migrated to a standby node where it is not necessarily installed, fails and becomes unmanaged.

Expected results:

I expect the systemd high availability resource to be started on an online node where it is installed.

Additional info:

sudo crm_mon

Last updated: Thu Jan 21 14:50:21 2016          Last change: Thu Jan 21 14:07:28 2016 by hacluster via crmd on tauko
Stack: corosync
Current DC: tauti (version 1.1.13-10.el7-44eb2dd) - partition with quorum
3 nodes and 13 resources configured

Node tauti: standby
Node teema: standby
Online: [ tauko ]

DMS-IP  (ocf::heartbeat:IPaddr2):	Started tauko
 Resource Group: DMS
     apache2    (systemd:httpd):        Started tauko
     DMS-GW     (lsb:dms):	Started tauko
 Resource Group: PMC
     pmc-routing        (systemd:pmc-routing):  Started tauko
     pmc-email-amqp-dispatcher  (systemd:pmc-email-amqp-dispatcher):    Started tauko
     pmc-email-main     (systemd:pmc-email-main):	Started tauko
     pmc-smpp-receive-json	(systemd:pmc-smpp-receive-json):        Started tauko
     pmc-smpp-receive-dlr	(systemd:pmc-smpp-receive-dlr): Started tauko
     pmc-smpp-receive-msg	(systemd:pmc-smpp-receive-msg): Started tauko
     postfix    (systemd:postfix):	FAILED teema (unmanaged)
 Resource Group: kannel
     kannel-bearerbox   (systemd:kannel-bearerbox):     Started tauko
     kannel-smsbox	(systemd:kannel-smsbox):        Started tauko
     kannel-wapbox	(systemd:kannel-wapbox):        Started tauko

Failed Actions:
* postfix_stop_0 on teema 'unknown error' (1): call=-1, status=Timed Out, exitreason='none',
    last-rc-change='Thu Jan 21 14:10:08 2016', queued=0ms, exec=0ms

tauti /var/log/messages:
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_tra
nsition_graph ]
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti pengine[2144]: warning: Forcing pmc-routing away from tauko after 1000000 failures (max=1000000)
Jan 21 14:02:08 tauti pengine[2144]:  notice: Calculated Transition 98: /var/lib/pacemaker/pengine/pe-input-274.bz2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]:  notice: Notifications disabled
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:02:08 tauti pengine[2144]:  notice: Start   pmc-routing#011(tauko)
Jan 21 14:02:08 tauti pengine[2144]:  notice: Start   pmc-email-amqp-dispatcher#011(tauko)
Jan 21 14:02:08 tauti pengine[2144]:  notice: Start   pmc-email-main#011(tauko)
Jan 21 14:02:08 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-json#011(tauko)
Jan 21 14:02:08 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-dlr#011(tauko)
Jan 21 14:02:08 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-msg#011(tauko)
Jan 21 14:02:08 tauti pengine[2144]:  notice: Start   postfix#011(tauko)
Jan 21 14:02:08 tauti pengine[2144]:  notice: Calculated Transition 99: /var/lib/pacemaker/pengine/pe-input-275.bz2
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 26: monitor pmc-routing_monitor_0 on teema
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 18: monitor pmc-routing_monitor_0 on tauti (local)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 10: monitor pmc-routing_monitor_0 on tauko
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 27: monitor pmc-email-amqp-dispatcher_monitor_0 on teema
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 19: monitor pmc-email-amqp-dispatcher_monitor_0 on tauti (local)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 11: monitor pmc-email-amqp-dispatcher_monitor_0 on tauko
Jan 21 14:02:08 tauti crmd[2145]:  notice: Operation pmc-routing_monitor_0: not running (node=tauti, call=117, rc=7, cib-update=263, confirme
d=true)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Operation pmc-email-amqp-dispatcher_monitor_0: not running (node=tauti, call=121, rc=7, cib-update
=264, confirmed=true)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 28: monitor pmc-email-main_monitor_0 on teema
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 20: monitor pmc-email-main_monitor_0 on tauti (local)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 21: monitor pmc-smpp-receive-json_monitor_0 on tauti (local)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Operation pmc-email-main_monitor_0: not running (node=tauti, call=125, rc=7, cib-update=265, confi
rmed=true)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Operation pmc-smpp-receive-json_monitor_0: not running (node=tauti, call=129, rc=7, cib-update=266
, confirmed=true)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 12: monitor pmc-email-main_monitor_0 on tauko
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 22: monitor pmc-smpp-receive-dlr_monitor_0 on tauti (local)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 23: monitor pmc-smpp-receive-msg_monitor_0 on tauti (local)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Operation pmc-smpp-receive-dlr_monitor_0: not running (node=tauti, call=133, rc=7, cib-update=267,
 confirmed=true)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Operation pmc-smpp-receive-msg_monitor_0: not running (node=tauti, call=137, rc=7, cib-update=268,
 confirmed=true)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 29: monitor pmc-smpp-receive-json_monitor_0 on teema
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 13: monitor pmc-smpp-receive-json_monitor_0 on tauko
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 30: monitor pmc-smpp-receive-dlr_monitor_0 on teema
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 14: monitor pmc-smpp-receive-dlr_monitor_0 on tauko
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 24: monitor postfix_monitor_0 on tauti (local)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Operation postfix_monitor_0: not running (node=tauti, call=141, rc=7, cib-update=269, confirmed=tr
ue)
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 31: monitor pmc-smpp-receive-msg_monitor_0 on teema
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 15: monitor pmc-smpp-receive-msg_monitor_0 on tauko
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 32: monitor postfix_monitor_0 on teema
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 17: probe_complete probe_complete-tauti on tauti (local) - no waiting
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 16: monitor postfix_monitor_0 on tauko
Jan 21 14:02:08 tauti crmd[2145]:  notice: Initiating action 9: probe_complete probe_complete-tauko on tauko - no waiting
...
Jan 21 14:03:28 tauti crmd[2145]: warning: Timer popped (timeout=20000, abort_level=0, complete=false)
Jan 21 14:03:28 tauti crmd[2145]:   error: [Action   32]: In-flight rsc op postfix_monitor_0                 on teema (priority: 0, waiting: 
none)
Jan 21 14:03:28 tauti crmd[2145]:  notice: Transition aborted: Action lost (source=action_timer_callback:805, 0)
Jan 21 14:03:28 tauti crmd[2145]: warning: rsc_op 32: postfix_monitor_0 on teema timed out
Jan 21 14:03:28 tauti crmd[2145]:  notice: Initiating action 25: probe_complete probe_complete-teema on teema - no waiting
Jan 21 14:03:28 tauti crmd[2145]:  notice: Transition 99 (Complete=26, Pending=0, Fired=0, Skipped=1, Incomplete=15, Source=/var/lib/pacemake
r/pengine/pe-input-275.bz2): Stopped
Jan 21 14:03:28 tauti pengine[2144]: warning: Processing failed op monitor for postfix on teema: unknown error (1)
Jan 21 14:03:28 tauti pengine[2144]: warning: Processing failed op monitor for postfix on teema: unknown error (1)
Jan 21 14:03:28 tauti pengine[2144]:   error: postfix[tauko] = 1000000
Jan 21 14:03:28 tauti pengine[2144]:   error: postfix[tauti] = 1000000
Jan 21 14:03:28 tauti pengine[2144]:  notice: Start   pmc-routing#011(tauko)
Jan 21 14:03:28 tauti pengine[2144]:  notice: Start   pmc-email-amqp-dispatcher#011(tauko)
Jan 21 14:03:28 tauti pengine[2144]:  notice: Start   pmc-email-main#011(tauko)
Jan 21 14:03:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-json#011(tauko)
Jan 21 14:03:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-dlr#011(tauko)
Jan 21 14:03:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-msg#011(tauko)
Jan 21 14:03:28 tauti pengine[2144]:  notice: Recover postfix#011(Started teema -> tauko)
Jan 21 14:03:28 tauti crmd[2145]:  notice: Initiating action 1: stop postfix_stop_0 on teema
Jan 21 14:03:28 tauti pengine[2144]:  notice: Calculated Transition 100: /var/lib/pacemaker/pengine/pe-input-276.bz2
...
Jan 21 14:04:48 tauti crmd[2145]: warning: Timer popped (timeout=20000, abort_level=0, complete=false)
Jan 21 14:04:48 tauti crmd[2145]:   error: [Action    1]: In-flight rsc op postfix_stop_0                    on teema (priority: 0, waiting: 
none)
Jan 21 14:04:48 tauti crmd[2145]:  notice: Transition aborted: Action lost (source=action_timer_callback:805, 0)
Jan 21 14:04:48 tauti crmd[2145]: warning: rsc_op 1: postfix_stop_0 on teema timed out
Jan 21 14:04:48 tauti crmd[2145]:  notice: Transition 100 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=18, Source=/var/lib/pacemaker/pengine/pe-input-276.bz2): Complete
Jan 21 14:04:49 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:04:49 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-routing#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-email-amqp-dispatcher#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-email-main#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-json#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-dlr#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-msg#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Calculated Transition 101: /var/lib/pacemaker/pengine/pe-input-277.bz2
Jan 21 14:04:49 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:04:49 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:04:49 tauti pengine[2144]: warning: Forcing postfix away from teema after 1000000 failures (max=1000000)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-routing#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-email-amqp-dispatcher#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-email-main#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-json#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-dlr#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-msg#011(tauko)
Jan 21 14:04:49 tauti pengine[2144]:  notice: Calculated Transition 102: /var/lib/pacemaker/pengine/pe-input-278.bz2
Jan 21 14:04:49 tauti crmd[2145]:  notice: Initiating action 22: start pmc-routing_start_0 on tauko
Jan 21 14:04:51 tauti crmd[2145]:  notice: Initiating action 23: monitor pmc-routing_monitor_60000 on tauko
Jan 21 14:04:51 tauti crmd[2145]:  notice: Initiating action 24: start pmc-email-amqp-dispatcher_start_0 on tauko
Jan 21 14:04:53 tauti crmd[2145]:  notice: Initiating action 25: monitor pmc-email-amqp-dispatcher_monitor_60000 on tauko
Jan 21 14:04:53 tauti crmd[2145]:  notice: Initiating action 26: start pmc-email-main_start_0 on tauko
Jan 21 14:04:56 tauti crmd[2145]:  notice: Initiating action 27: monitor pmc-email-main_monitor_60000 on tauko
Jan 21 14:04:56 tauti crmd[2145]:  notice: Initiating action 28: start pmc-smpp-receive-json_start_0 on tauko
Jan 21 14:04:58 tauti crmd[2145]:  notice: Initiating action 29: monitor pmc-smpp-receive-json_monitor_60000 on tauko
Jan 21 14:04:58 tauti crmd[2145]:  notice: Initiating action 30: start pmc-smpp-receive-dlr_start_0 on tauko
Jan 21 14:05:00 tauti crmd[2145]:  notice: Initiating action 31: monitor pmc-smpp-receive-dlr_monitor_60000 on tauko
Jan 21 14:05:00 tauti crmd[2145]:  notice: Initiating action 32: start pmc-smpp-receive-msg_start_0 on taiko
...
Jan 21 14:05:02 tauti crmd[2145]:  notice: Initiating action 33: monitor pmc-smpp-receive-msg_monitor_60000 on tauko
Jan 21 14:05:02 tauti crmd[2145]:  notice: Transition 102 (Complete=13, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemake
r/pengine/pe-input-278.bz2): Complete
Jan 21 14:05:02 tauti crmd[2145]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=no
tify_crmd ]
...
Jan 21 14:07:27 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:27 tauti crmd[2145]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_tra
nsition_graph ]
Jan 21 14:07:27 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:27 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:27 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:27 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:27 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:27 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:27 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:27 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:07:27 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:07:27 tauti pengine[2144]: warning: Forcing postfix away from teema after 1000000 failures (max=1000000)
Jan 21 14:07:27 tauti pengine[2144]:  notice: Calculated Transition 103: /var/lib/pacemaker/pengine/pe-input-279.bz2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]:  notice: Notifications disabled
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]:  notice: Notifications disabled
Jan 21 14:07:28 tauti crmd[2145]:  notice: Notifications disabled
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]:  notice: Notifications disabled
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 1
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 3
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti crmd[2145]: warning: No match for shutdown action on 2
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-routing#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-email-amqp-dispatcher#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-email-main#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-json#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-dlr#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-msg#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   postfix#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Calculated Transition 104: /var/lib/pacemaker/pengine/pe-input-280.bz2
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 26: monitor pmc-routing_monitor_0 on teema
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 18: monitor pmc-routing_monitor_0 on tauti (local)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 10: monitor pmc-routing_monitor_0 on tauko
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 27: monitor pmc-email-amqp-dispatcher_monitor_0 on teema
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 19: monitor pmc-email-amqp-dispatcher_monitor_0 on tauti (local)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 11: monitor pmc-email-amqp-dispatcher_monitor_0 on tauko
Jan 21 14:07:28 tauti crmd[2145]:  notice: Operation pmc-routing_monitor_0: not running (node=tauti, call=152, rc=7, cib-update=360, confirme
d=true)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Operation pmc-email-amqp-dispatcher_monitor_0: not running (node=tauti, call=156, rc=7, cib-update
=361, confirmed=true)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 28: monitor pmc-email-main_monitor_0 on teema
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 20: monitor pmc-email-main_monitor_0 on tauti (local)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 29: monitor pmc-smpp-receive-json_monitor_0 on teema
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 21: monitor pmc-smpp-receive-json_monitor_0 on tauti (local)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Operation pmc-email-main_monitor_0: not running (node=tauti, call=160, rc=7, cib-update=362, confi
rmed=true)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Operation pmc-smpp-receive-json_monitor_0: not running (node=tauti, call=164, rc=7, cib-update=363
, confirmed=true)
Jan 21 14:07:28 tauti crmd[2145]: warning: Action 10 (pmc-routing_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:07:28 tauti crmd[2145]:  notice: Transition aborted by pmc-routing_monitor_0 'create' on tauko: Event failed (magic=0:0;10:104:7:da
931aba-558d-4290-a05b-6f5971f308e0, cib=0.296.19, source=match_graph_event:381, 0)
Jan 21 14:07:28 tauti crmd[2145]: warning: Action 10 (pmc-routing_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:07:28 tauti crmd[2145]: warning: Action 11 (pmc-email-amqp-dispatcher_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:07:28 tauti crmd[2145]: warning: Action 11 (pmc-email-amqp-dispatcher_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:07:28 tauti crmd[2145]:  notice: Transition 104 (Complete=11, Pending=0, Fired=0, Skipped=11, Incomplete=30, Source=/var/lib/pacema
ker/pengine/pe-input-280.bz2): Stopped
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-email-main#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-json#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-dlr#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-msg#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Start   postfix#011(tauko)
Jan 21 14:07:28 tauti pengine[2144]:  notice: Calculated Transition 105: /var/lib/pacemaker/pengine/pe-input-281.bz2
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 35: monitor pmc-routing_monitor_60000 on tauko
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 38: monitor pmc-email-amqp-dispatcher_monitor_60000 on tauko
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 20: monitor pmc-smpp-receive-dlr_monitor_0 on teema
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 16: monitor pmc-smpp-receive-dlr_monitor_0 on tauti (local)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 21: monitor pmc-smpp-receive-msg_monitor_0 on teema
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 17: monitor pmc-smpp-receive-msg_monitor_0 on tauti (local)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Operation pmc-smpp-receive-dlr_monitor_0: not running (node=tauti, call=168, rc=7, cib-update=365, confirmed=true)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Operation pmc-smpp-receive-msg_monitor_0: not running (node=tauti, call=172, rc=7, cib-update=366, confirmed=true)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 10: monitor pmc-email-main_monitor_0 on tauko
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 11: monitor pmc-smpp-receive-json_monitor_0 on tauko
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 22: monitor postfix_monitor_0 on teema
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 18: monitor postfix_monitor_0 on tauti (local)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Operation postfix_monitor_0: not running (node=tauti, call=176, rc=7, cib-update=367, confirmed=true)
Jan 21 14:07:28 tauti crmd[2145]:  notice: Initiating action 15: probe_complete probe_complete-tauti on tauti (local) - no waiting
Jan 21 14:07:28 tauti crmd[2145]: warning: Action 10 (pmc-email-main_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:07:28 tauti crmd[2145]:  notice: Transition aborted by pmc-email-main_monitor_0 'create' on tauko: Event failed (magic=0:0;10:105:7:da931aba-558d-4290-a05b-6f5971f308e0, cib=0.296.32, source=match_graph_event:381, 0)
Jan 21 14:07:28 tauti crmd[2145]: warning: Action 10 (pmc-email-main_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:07:28 tauti crmd[2145]: warning: Action 11 (pmc-smpp-receive-json_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:07:28 tauti crmd[2145]: warning: Action 11 (pmc-smpp-receive-json_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
...
Jan 21 14:08:48 tauti crmd[2145]: warning: Timer popped (timeout=20000, abort_level=1, complete=false)
Jan 21 14:08:48 tauti crmd[2145]:   error: [Action   22]: In-flight rsc op postfix_monitor_0                 on teema (priority: 0, waiting: 
none)
Jan 21 14:08:48 tauti crmd[2145]:  notice: Transition aborted: Action lost (source=action_timer_callback:805, 0)
Jan 21 14:08:48 tauti crmd[2145]: warning: rsc_op 22: postfix_monitor_0 on teema timed out
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 19: probe_complete probe_complete-teema on teema - no waiting
Jan 21 14:08:48 tauti crmd[2145]:  notice: Transition 105 (Complete=13, Pending=0, Fired=0, Skipped=3, Incomplete=16, Source=/var/lib/pacemak
er/pengine/pe-input-281.bz2): Stopped
Jan 21 14:08:48 tauti pengine[2144]: warning: Processing failed op monitor for postfix on teema: unknown error (1)
Jan 21 14:08:48 tauti pengine[2144]: warning: Processing failed op monitor for postfix on teema: unknown error (1)
Jan 21 14:08:48 tauti pengine[2144]:   error: postfix[tauko] = 1000000
Jan 21 14:08:48 tauti pengine[2144]:   error: postfix[tauti] = 1000000
Jan 21 14:08:48 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-dlr#011(tauko)
Jan 21 14:08:48 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-msg#011(tauko)
Jan 21 14:08:48 tauti pengine[2144]:  notice: Recover postfix#011(Started teema -> tauko)
Jan 21 14:08:48 tauti pengine[2144]:  notice: Calculated Transition 106: /var/lib/pacemaker/pengine/pe-input-282.bz2
Jan 21 14:08:48 tauti pengine[2144]: warning: Processing failed op monitor for postfix on teema: unknown error (1)
Jan 21 14:08:48 tauti pengine[2144]: warning: Processing failed op monitor for postfix on teema: unknown error (1)
Jan 21 14:08:48 tauti pengine[2144]:   error: postfix[tauko] = 1000000
Jan 21 14:08:48 tauti pengine[2144]:   error: postfix[tauti] = 1000000
Jan 21 14:08:48 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-dlr#011(tauko)
Jan 21 14:08:48 tauti pengine[2144]:  notice: Start   pmc-smpp-receive-msg#011(tauko)
Jan 21 14:08:48 tauti pengine[2144]:  notice: Recover postfix#011(Started teema -> tauko)
Jan 21 14:08:48 tauti pengine[2144]:  notice: Calculated Transition 107: /var/lib/pacemaker/pengine/pe-input-282.bz2
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 34: monitor pmc-email-main_monitor_60000 on tauko
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 37: monitor pmc-smpp-receive-json_monitor_60000 on tauko
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 13: monitor pmc-smpp-receive-dlr_monitor_0 on tauko
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 14: monitor pmc-smpp-receive-msg_monitor_0 on tauko
Jan 21 14:08:48 tauti crmd[2145]: warning: Action 13 (pmc-smpp-receive-dlr_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:08:48 tauti crmd[2145]:  notice: Transition aborted by pmc-smpp-receive-dlr_monitor_0 'create' on tauko: Event failed (magic=0:0;13
:107:7:da931aba-558d-4290-a05b-6f5971f308e0, cib=0.296.37, source=match_graph_event:381, 0)
Jan 21 14:08:48 tauti crmd[2145]: warning: Action 13 (pmc-smpp-receive-dlr_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:08:48 tauti crmd[2145]: warning: Action 14 (pmc-smpp-receive-msg_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:08:48 tauti crmd[2145]: warning: Action 14 (pmc-smpp-receive-msg_monitor_0) on tauko failed (target: 7 vs. rc: 0): Error
Jan 21 14:08:48 tauti crmd[2145]:  notice: Transition 107 (Complete=5, Pending=0, Fired=0, Skipped=1, Incomplete=14, Source=/var/lib/pacemake
r/pengine/pe-input-282.bz2): Stopped
Jan 21 14:08:48 tauti pengine[2144]: warning: Processing failed op monitor for postfix on teema: unknown error (1)
Jan 21 14:08:48 tauti pengine[2144]: warning: Processing failed op monitor for postfix on teema: unknown error (1)
Jan 21 14:08:48 tauti pengine[2144]:   error: postfix[tauko] = 1000000
Jan 21 14:08:48 tauti pengine[2144]:   error: postfix[tauti] = 1000000
Jan 21 14:08:48 tauti pengine[2144]:  notice: Recover postfix#011(Started teema -> tauko)
Jan 21 14:08:48 tauti pengine[2144]:  notice: Calculated Transition 108: /var/lib/pacemaker/pengine/pe-input-283.bz2
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 38: monitor pmc-smpp-receive-dlr_monitor_60000 on tauko
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 41: monitor pmc-smpp-receive-msg_monitor_60000 on tauko
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 15: monitor postfix_monitor_0 on tauko
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 14: probe_complete probe_complete-tauko on tauko - no waiting
Jan 21 14:08:48 tauti crmd[2145]:  notice: Initiating action 1: stop postfix_stop_0 on teema
...
Jan 21 14:10:08 tauti crmd[2145]: warning: Timer popped (timeout=20000, abort_level=0, complete=false)
Jan 21 14:10:08 tauti crmd[2145]:   error: [Action    1]: In-flight rsc op postfix_stop_0                    on teema (priority: 0, waiting: 
none)
Jan 21 14:10:08 tauti crmd[2145]:  notice: Transition aborted: Action lost (source=action_timer_callback:805, 0)
Jan 21 14:10:08 tauti crmd[2145]: warning: rsc_op 1: postfix_stop_0 on teema timed out
Jan 21 14:10:08 tauti crmd[2145]:  notice: Transition 108 (Complete=7, Pending=0, Fired=0, Skipped=0, Incomplete=6, Source=/var/lib/pacemaker
/pengine/pe-input-283.bz2): Complete
Jan 21 14:10:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:10:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:10:09 tauti pengine[2144]:  notice: Calculated Transition 109: /var/lib/pacemaker/pengine/pe-input-284.bz2
Jan 21 14:10:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:10:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:10:09 tauti pengine[2144]: warning: Forcing postfix away from teema after 1000000 failures (max=1000000)
Jan 21 14:10:09 tauti pengine[2144]:  notice: Calculated Transition 110: /var/lib/pacemaker/pengine/pe-input-285.bz2
Jan 21 14:10:09 tauti crmd[2145]:  notice: Transition 110 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-285.bz2): Complete
Jan 21 14:10:09 tauti crmd[2145]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
...
Jan 21 14:25:09 tauti crmd[2145]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer
_popped ]
Jan 21 14:25:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:25:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:25:09 tauti pengine[2144]: warning: Forcing postfix away from teema after 1000000 failures (max=1000000)
Jan 21 14:25:09 tauti pengine[2144]:  notice: Calculated Transition 111: /var/lib/pacemaker/pengine/pe-input-285.bz2
Jan 21 14:25:09 tauti crmd[2145]:  notice: Transition 111 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker
/pengine/pe-input-285.bz2): Complete
Jan 21 14:25:09 tauti crmd[2145]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=no
tify_crmd ]
...
Jan 21 14:40:09 tauti crmd[2145]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer
_popped ]
Jan 21 14:40:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:40:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:40:09 tauti pengine[2144]: warning: Forcing postfix away from teema after 1000000 failures (max=1000000)
Jan 21 14:40:09 tauti pengine[2144]:  notice: Calculated Transition 112: /var/lib/pacemaker/pengine/pe-input-285.bz2
Jan 21 14:40:09 tauti crmd[2145]:  notice: Transition 112 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker
/pengine/pe-input-285.bz2): Complete
Jan 21 14:40:09 tauti crmd[2145]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=no
tify_crmd ]
...
Jan 21 14:40:09 tauti crmd[2145]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer
_popped ]
Jan 21 14:40:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:40:09 tauti pengine[2144]: warning: Processing failed op stop for postfix on teema: unknown error (1)
Jan 21 14:40:09 tauti pengine[2144]: warning: Forcing postfix away from teema after 1000000 failures (max=1000000)
Jan 21 14:40:09 tauti pengine[2144]:  notice: Calculated Transition 112: /var/lib/pacemaker/pengine/pe-input-285.bz2
Jan 21 14:40:09 tauti crmd[2145]:  notice: Transition 112 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-285.bz2): Complete
Jan 21 14:40:09 tauti crmd[2145]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]

teema /var/log/messages:
Jan 21 14:02:08 localhost crmd[1627]:  notice: Operation pmc-routing_monitor_0: not running (node=teema, call=65, rc=7, cib-update=55, confirmed=true)
Jan 21 14:02:08 localhost crmd[1627]:  notice: Operation pmc-email-amqp-dispatcher_monitor_0: not running (node=teema, call=69, rc=7, cib-update=56, confirmed=true)
Jan 21 14:02:08 localhost crmd[1627]:  notice: Operation pmc-email-main_monitor_0: not running (node=teema, call=73, rc=7, cib-update=57, confirmed=true)
Jan 21 14:02:08 localhost crmd[1627]:  notice: Operation pmc-smpp-receive-json_monitor_0: not running (node=teema, call=77, rc=7, cib-update=58, confirmed=true)
Jan 21 14:02:08 localhost crmd[1627]:  notice: Operation pmc-smpp-receive-dlr_monitor_0: not running (node=teema, call=81, rc=7, cib-update=59, confirmed=true)
Jan 21 14:02:08 localhost crmd[1627]:  notice: Operation pmc-smpp-receive-msg_monitor_0: not running (node=teema, call=85, rc=7, cib-update=60, confirmed=true)
Jan 21 14:07:28 localhost crmd[1627]:   error: Op postfix_stop_0 (call=90): Cancelled
Jan 21 14:07:28 localhost crmd[1627]:  notice: Operation pmc-routing_monitor_0: not running (node=teema, call=102, rc=7, cib-update=90, confirmed=true)
Jan 21 14:07:28 localhost crmd[1627]:  notice: Operation pmc-email-amqp-dispatcher_monitor_0: not running (node=teema, call=106, rc=7, cib-update=91, confirmed=true)
Jan 21 14:07:28 localhost crmd[1627]:  notice: Operation pmc-email-main_monitor_0: not running (node=teema, call=110, rc=7, cib-update=92, confirmed=true)
Jan 21 14:07:28 localhost crmd[1627]:  notice: Operation pmc-smpp-receive-json_monitor_0: not running (node=teema, call=114, rc=7, cib-update=93, confirmed=true)
Jan 21 14:07:28 localhost crmd[1627]:  notice: Operation pmc-smpp-receive-dlr_monitor_0: not running (node=teema, call=118, rc=7, cib-update=94, confirmed=true)
Jan 21 14:07:28 localhost crmd[1627]:  notice: Operation pmc-smpp-receive-msg_monitor_0: not running (node=teema, call=122, rc=7, cib-update=95, confirmed=true)
Comment 2 Ken Gaillot 2016-01-21 10:02:31 EST
This is configurable in Pacemaker, using one of these methods:

* Usually, the simplest is to configure what nodes the resource is not allowed to run on. For example, "pcs constraint location postfix avoids teema".

* If there are many such cases, it may be simpler to do the reverse, and specify what nodes a resource is allowed to run on. This is done by making the cluster opt-in ("pcs property set symmetric-cluster=false") and then setting positive location preferences for all resources on all nodes they can run on (for example, "pcs constraint location postfix prefers tauko=100; pcs constraint location postfix prefers tauti=100").
Comment 3 Matti Linnanvuori 2016-01-22 02:17:00 EST
symmetric-cluster property is already off. There are infinite location preferences for tauko and tauti but Pacemaker moves resources to teema anyway.

sudo pcs config
Cluster Name: MDCS
Corosync Nodes:
 tauko tauti teema 
Pacemaker Nodes:
 tauko tauti teema 

Resources: 
 Resource: DMS-IP (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=10.80.7.107 
  Meta Attrs: resource-stickiness=INFINITY 
  Operations: start interval=0s timeout=20s (DMS-IP-start-interval-0s)
              stop interval=0s timeout=20s (DMS-IP-stop-interval-0s)
              monitor interval=10s timeout=20s (DMS-IP-monitor-interval-10s)
 Group: DMS
  Resource: apache2 (class=systemd type=httpd)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (apache2-monitor-interval-60s)
  Resource: DMS-GW (class=lsb type=dms)
   Operations: monitor interval=60s (DMS-GW-monitor-interval-60s)
 Group: PMC
  Resource: pmc-routing (class=systemd type=pmc-routing)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (pmc-routing-monitor-interval-60s)
  Resource: pmc-email-amqp-dispatcher (class=systemd type=pmc-email-amqp-dispatcher)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (pmc-email-amqp-dispatcher-monitor-interval-60s)
  Resource: pmc-email-main (class=systemd type=pmc-email-main)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (pmc-email-main-monitor-interval-60s)
  Resource: pmc-smpp-receive-json (class=systemd type=pmc-smpp-receive-json)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (pmc-smpp-receive-json-monitor-interval-60s)
  Resource: pmc-smpp-receive-dlr (class=systemd type=pmc-smpp-receive-dlr)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (pmc-smpp-receive-dlr-monitor-interval-60s)
  Resource: pmc-smpp-receive-msg (class=systemd type=pmc-smpp-receive-msg)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (pmc-smpp-receive-msg-monitor-interval-60s)
  Resource: postfix (class=systemd type=postfix)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (postfix-monitor-interval-60s)
 Group: kannel
  Resource: kannel-bearerbox (class=systemd type=kannel-bearerbox)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (kannel-bearerbox-monitor-interval-60s)
  Resource: kannel-smsbox (class=systemd type=kannel-smsbox)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (kannel-smsbox-monitor-interval-60s)
  Resource: kannel-wapbox (class=systemd type=kannel-wapbox)
   Meta Attrs: resource-stickiness=INFINITY 
   Operations: monitor interval=60s (kannel-wapbox-monitor-interval-60s)

Stonith Devices: 
Fencing Levels: 

Location Constraints:
  Resource: DMS-GW
    Enabled on: tauko (score:INFINITY) (id:location-DMS-GW-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-DMS-GW-tauti-INFINITY)
  Resource: DMS-IP
    Enabled on: tauko (score:INFINITY) (id:location-DMS-IP-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-DMS-IP-tauti-INFINITY)
  Resource: apache2
    Enabled on: tauko (score:INFINITY) (id:location-apache2-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-apache2-tauti-INFINITY)
  Resource: kannel-bearerbox
    Enabled on: tauko (score:INFINITY) (id:location-kannel-bearerbox-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-kannel-bearerbox-tauti-INFINITY)
  Resource: kannel-smsbox
    Enabled on: tauko (score:INFINITY) (id:location-kannel-smsbox-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-kannel-smsbox-tauti-INFINITY)
  Resource: kannel-wapbox
    Enabled on: tauko (score:INFINITY) (id:location-kannel-wapbox-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-kannel-wapbox-tauti-INFINITY)
  Resource: pmc-email-amqp-dispatcher
    Enabled on: tauko (score:INFINITY) (id:location-pmc-email-amqp-dispatcher-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-pmc-email-amqp-dispatcher-tauti-INFINITY)
  Resource: pmc-email-main
    Enabled on: tauko (score:INFINITY) (id:location-pmc-email-main-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-pmc-email-main-tauti-INFINITY)
  Resource: pmc-routing
    Enabled on: tauko (score:INFINITY) (id:location-pmc-routing-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-pmc-routing-tauti-INFINITY)
  Resource: pmc-smpp-receive-dlr
    Enabled on: tauko (score:INFINITY) (id:location-pmc-smpp-receive-dlr-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-pmc-smpp-receive-dlr-tauti-INFINITY)
  Resource: pmc-smpp-receive-json
    Enabled on: tauko (score:INFINITY) (id:location-pmc-smpp-receive-json-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-pmc-smpp-receive-json-tauti-INFINITY)
  Resource: pmc-smpp-receive-msg
    Enabled on: tauko (score:INFINITY) (id:location-pmc-smpp-receive-msg-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-pmc-smpp-receive-msg-tauti-INFINITY)
  Resource: postfix
    Enabled on: tauko (score:INFINITY) (id:location-postfix-tauko-INFINITY)
    Enabled on: tauti (score:INFINITY) (id:location-postfix-tauti-INFINITY)
Ordering Constraints:
  start apache2 then start DMS-GW (score:INFINITY) (id:order-apache2-DMS-GW-INFINITY)
  stop DMS-GW then stop apache2 (score:INFINITY) (id:order-DMS-GW-apache2-INFINITY)
Colocation Constraints:
  apache2 with DMS-IP (score:INFINITY) (id:colocation-apache2-DMS-IP-INFINITY)
  DMS-GW with apache2 (score:INFINITY) (id:colocation-DMS-GW-apache2-INFINITY)
  pmc-email-amqp-dispatcher with pmc-routing (score:INFINITY) (id:colocation-pmc-email-amqp-dispatcher-pmc-routing-INFINITY)
  pmc-email-main with pmc-routing (score:INFINITY) (id:colocation-pmc-email-main-pmc-routing-INFINITY)
  pmc-smpp-receive-dlr with pmc-routing (score:INFINITY) (id:colocation-pmc-smpp-receive-dlr-pmc-routing-INFINITY)
  pmc-smpp-receive-json with pmc-routing (score:INFINITY) (id:colocation-pmc-smpp-receive-json-pmc-routing-INFINITY)
  pmc-smpp-receive-msg with pmc-routing (score:INFINITY) (id:colocation-pmc-smpp-receive-msg-pmc-routing-INFINITY)
  postfix with pmc-email-amqp-dispatcher (score:INFINITY) (id:colocation-postfix-pmc-email-amqp-dispatcher-INFINITY)
  kannel-bearerbox with DMS-IP (score:INFINITY) (id:colocation-kannel-bearerbox-DMS-IP-INFINITY)
  kannel-smsbox with kannel-bearerbox (score:INFINITY) (id:colocation-kannel-smsbox-kannel-bearerbox-INFINITY)
  kannel-wapbox with kannel-bearerbox (score:INFINITY) (id:colocation-kannel-wapbox-kannel-bearerbox-INFINITY)
  pmc-routing with kannel-smsbox (score:INFINITY) (id:colocation-pmc-routing-kannel-smsbox-INFINITY)

Resources Defaults:
 No defaults set
Operations Defaults:

 No defaults set

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: MDCS
 dc-version: 1.1.13-10.el7-44eb2dd
 have-watchdog: false
 last-lrm-refresh: 1453387160
 maintenance-mode: false
 no-quorum-policy: stop
 stonith-enabled: off
 symmetric-cluster: off
Node Attributes:
 tauti: standby=on
 teema: standby=on
Comment 4 Ken Gaillot 2016-01-22 12:34:02 EST
Matti,

Thanks for the additional details.

I'm not sure this is causing problems, but scores less than INFINITY should be used when enabling resources on nodes. INFINITY means mandatory, so it's effectively saying the resource must run in two places. If you use for example 100, it would allow (but not require) the resource to run on the node.

However, the main issue is that, even if a resource is not allowed to run on a node, pacemaker will attempt to probe the resource on that node, to ensure that it is indeed not running there. Since postfix isn't installed on teema, that probe is failing, so pacemaker is considering it possibly already started there.

You could install (but not enable) postfix on teema so that pacemaker can verify that it's not running. (Odd, I know, but because it's a systemd resource, systemd needs the unit file to report that it's not running.)

Alternatively, you can use the ocf:heartbeat:postfix resource agent instead of systemd:postfix. I believe the OCF agent will correctly report that postfix is not installed.
Comment 6 Matti Linnanvuori 2016-01-25 07:30:55 EST
postfix is installed on teema. It did not seem to matter if a resource was installed or not on teema when I got the same error with resources other than postfix. Changing the location constraints' score from INFINITY to 100 did not help. Changing the resource agent to ocf::heartbeat:postfix did not help either: I still got the same error:

Last updated: Mon Jan 25 14:28:09 2016          Last change: Mon Jan 25 13:20:56 2016 by hacluster via crmd on tauko
Stack: corosync
Current DC: tauti (version 1.1.13-10.el7-44eb2dd) - partition with quorum
3 nodes and 16 resources configured

Node teema: standby
Online: [ tauko tauti ]

DMS-IP  (ocf::heartbeat:IPaddr2):	Started tauko
 Resource Group: DMS
     apache2    (systemd:httpd):        Started tauko
     DMS-GW     (lsb:dms):	Started tauko
 Resource Group: PMC
     pmc-routing        (systemd:pmc-routing):  Started tauko
     pmc-email-amqp-dispatcher  (systemd:pmc-email-amqp-dispatcher):    Started tauko
     pmc-email-main     (systemd:pmc-email-main):	Started tauko
     pmc-smpp-receive-json	(systemd:pmc-smpp-receive-json):        Started tauko
     pmc-smpp-receive-dlr	(systemd:pmc-smpp-receive-dlr): Started tauko
     pmc-smpp-receive-msg	(systemd:pmc-smpp-receive-msg): Started tauko
     pmc-astrid-112     (systemd:pmc-astrid-112):	Started tauko
     pmc-astrid-112-conftool    (systemd:pmc-astrid-112-conftool):	Started tauko
     pmc-astrid-112-conftool-celery     (systemd:pmc-astrid-112-conftool-celery):	Started tauko
     postfix    (ocf::heartbeat:postfix):	FAILED teema (unmanaged)
 Resource Group: kannel
     kannel-bearerbox   (systemd:kannel-bearerbox):     Started tauko
     kannel-smsbox	(systemd:kannel-smsbox):        Started tauko
     kannel-wapbox	(systemd:kannel-wapbox):        Started tauko

Failed Actions:
* postfix_stop_0 on teema 'unknown error' (1): call=-1, status=Timed Out, exitreason='none',
    last-rc-change='Mon Jan 25 13:23:49 2016', queued=0ms, exec=0ms
Comment 7 Klaus Wenninger 2016-01-25 08:18:43 EST
That the status of postfix is "Timed Out" is a little bit strange.
Die you try something like "systemctl status postfix" on teema if it has
postfix installed anyway?
Comment 8 Matti Linnanvuori 2016-01-25 08:45:57 EST
Yes, I tried systemctl status postfix. It quickly gave the following output:

systemctl status postfix
● postfix.service - Postfix Mail Transport Agent
   Loaded: loaded (/usr/lib/systemd/system/postfix.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Jan 20 16:07:26 teema systemd[1]: Starting Postfix Mail Transport Agent...
Jan 20 16:07:27 teema postfix/master[1582]: daemon started -- version 2.10.1, configuration /etc/postfix
Jan 20 16:07:27 teema systemd[1]: Started Postfix Mail Transport Agent.
Jan 20 16:07:52 teema systemd[1]: Stopping Postfix Mail Transport Agent...
Jan 20 16:07:52 teema systemd[1]: Stopped Postfix Mail Transport Agent.
Comment 9 Klaus Wenninger 2016-01-25 09:55:07 EST
Btw. when testing with the OCF-RA one would probably have to configure
systemd to keep it's hands off postfix. (delete the unit-file, overlay it ...)
Comment 10 Matti Linnanvuori 2016-01-25 10:25:25 EST
I removed the postfix unit files and executed systemctl daemon-reload. I got the same error again:

     postfix    (ocf::heartbeat:postfix):	FAILED teema (unmanaged)
...
Failed Actions:
* postfix_stop_0 on teema 'unknown error' (1): call=-1, status=Timed Out, exitreason='none',
    last-rc-change='Mon Jan 25 17:21:11 2016', queued=0ms, exec=0ms
Comment 11 Ken Gaillot 2016-01-25 15:30:24 EST
I didn't realize postfix was already installed on teema, with systemctl reporting the correct status. Can you reproduce the issue, run crm_report from around that time, and attach the result? I'm hoping the more detailed logs from that will shed more light on this.

FYI systemctl disable should be sufficient when using the OCF resource agent.
Comment 12 Matti Linnanvuori 2016-01-26 02:12 EST
Created attachment 1118383 [details]
crm_report output
Comment 13 Ken Gaillot 2016-01-26 13:47:17 EST
You may be able to simplify your configuration significantly:

* A group is essentially a shortcut for order and colocation constraints between its members, so you do not need to configure such constraints explicitly. For example, you don't need the "DMS-GW with apache2" or "apache2 then DMS-GW" constraints, because those are implied by their membership in the DMS group. The order that resources are listed in a group is the order in which they will be started, and all the resources in the group will be kept together on the same node.

* You can refer to a group in constraints, and the constraint will apply to all its members. For example, you could configure location constraints enabling the PMC group on tauko and tauti, so you wouldn't need separate such constraints for each resource in PMC.

* Order constraints normally imply their inverse. For example, if you have "start apache2 then DMS-GW", you don't need "stop DMS-GW then apache" because that is implied. (This is controlled by symmetrical=true in the constraint.)

I see you have stonith-enabled=false. Be aware that this prevents recovery from certain failure situations, and so clusters with stonith disabled are not supported by Red Hat. For details on configuring stonith, see https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/ch-fencing-HAAR.html

Back to the issue at hand, I am having trouble finding the cause. We see teema getting the probe request:

Jan 26 08:54:39 [1627] teema       crmd:     info: do_lrm_rsc_op:       Performing key=28:795:7:da931aba-558d-4290-a05b-6f5971f308e0 op=postfix_monitor_0

But then there are no further messages from crmd or lrmd on teema until after the DC (tauti) times out the operation.

Would you mind setting PCMK_debug=crmd,lrmd in /etc/sysconfig/pacemaker on all nodes, restarting the cluster, and repeating the test?
Comment 14 Matti Linnanvuori 2016-01-27 09:23:37 EST
I have been unable to reproduce the error with PCMK_debug=crmd,lrmd in /etc/sysconfig/pacemaker on all nodes and the cluster restarted.
Comment 15 Ken Gaillot 2016-01-27 11:22:23 EST
I don't think the debug setting could possibly matter here, so I'm guessing the cluster needed a full restart to clear some bad state. I will continue to investigate the previous logs to try to figure out what might have been wrong.

In the meantime, feel free to try the above configuration suggestions, and comment if the issue recurs.
Comment 16 Ken Gaillot 2016-01-27 13:13:52 EST
I should also have mentioned a new feature available since 1.1.13. It is now possible to specify the resource-discovery option in a location constraint to control whether the resource is probed on all nodes.

resource-discovery=always is the default; all nodes will be probed.

resource-discovery=never means do not probe the resource on the node specified in the location constraint.

resource-discovery=exclusive means probe the resource only on nodes that have this set.

Ideally, setting resource-discovery to never or exclusive should only be done if it is not possible to start the service on certain nodes (for example, the software is not installed), because this defeats pacemaker's ability to stop a rogue service where it's not supposed to be (for example, if an administrator started the service manually).
Comment 17 Matti Linnanvuori 2016-01-28 09:09 EST
Created attachment 1119172 [details]
crm_report output with debug logging
Comment 18 Matti Linnanvuori 2016-01-28 09:11:22 EST
I managed to reproduce the problem with debug logging on. I created a new attachment with a crm_report archive.
Comment 19 Ken Gaillot 2016-01-28 11:56:17 EST
I see one remaining issue with the constraints: DMS-IP is colocated with both kannel and PMC, but there is no constraint between kannel and PMC. So, the cluster could decide to put kannel and PMC on different nodes, in which case DMS-IP (and thus DMS) would be unable to run.

I would guess you want a colocation of kannel with PMC (or vice versa). If you want it to be able to run even if the other is not available, use a score less than INFINITY.

Looking at the new logs, I see two problems.

1. LRMD crashes:

Jan 28 15:51:30 [32526] teema pacemakerd:    error: child_waitpid:      Managed process 32597 (lrmd) dumped core

That did not happen in the original logs, so we may be looking at a separate problem. It occurred when the cluster was probing pmc-smpp-receive-dlr on teema (to ensure it's not running).

There is one known issue with lrmd crashing when reporting certain systemd errors, which will be fixed soon in an upcoming 7.2.z-stream release for BZ#1299339. Note that fix addresses a problem with logging the error, and is unrelated to whatever caused the error in the first place.

2. It looks like pmc-routing is now the resource that shows the original issue, and not postfix? The problem there looks similar. In this case, the cluster initiates a probe of pmc-routing on teema at 15:51:30, and teema successfully replies immediately with "not running". However, soon after that, the LRMD crash occurs, so the cluster initiates another probe 5 seconds later once it's back up. That probe times out; there is not a clear indication why, but I wouldn't be surprised if the LRMD crash and restart caused problems.

For now, I'd recommend setting resource-discovery=exclusive on all your location constraints. That will prevent the cluster from probing anything on teema, so we don't trigger that lrmd crash.

If you want to continue investigating, I'd recommend waiting for the 7.2.z to be released, apply that, then set resource-discovery back to always. Then, we can look at the logs again if the issue recurs.
Comment 20 Matti Linnanvuori 2016-01-29 03:35:27 EST
Yes, pmc-routing was the resource with the trouble last time. I noticed that probes timed out on the standby node teema also without a core dump.

I tried setting the colocation constraints' scores to 100 but then resource groups were allocated on different nodes, so I set the scores back to INFINITY.
Comment 21 Ken Gaillot 2016-02-18 16:18:40 EST
Matti,

The updated pacemaker packages fixing the lrmd crash have been released.

If everything is working with resource-discovery=exclusive, feel free to leave it like that, but if you want to try reproducing the error again, do a yum update and remove resource-discovery=exclusive.
Comment 22 Ken Gaillot 2016-07-05 16:57:43 EDT
I have not been able to determine a cause of the original issue yet, so a fix will not be ready in the 7.3 time frame. Moving to 7.4.

Matti, did resource-discovery=exclusive solve the issue for you? If you'd like to try to reproduce the issue again without resource-discovery=exclusive, let me know, and I can provide newer test packages.
Comment 23 Matti Linnanvuori 2016-07-25 03:59:03 EDT
resource-discovery=exclusive did solve the issue for me.
Comment 24 Ken Gaillot 2016-07-25 10:05:52 EDT
Good to hear, thanks. I'm closing this report, but feel free to reopen if you see the behavior again.

Note You need to log in before you can comment on or make changes to this bug.