RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1323546 - invalid transition when peer controlling a remote node dies
Summary: invalid transition when peer controlling a remote node dies
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pacemaker
Version: 7.2
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: 7.5
Assignee: Ken Gaillot
QA Contact: Ofer Blaut
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-04 02:27 UTC by Andrew Beekhof
Modified: 2018-04-10 15:29 UTC (History)
5 users (show)

Fixed In Version: pacemaker-1.1.18-1.el7
Doc Type: Bug Fix
Doc Text:
Cause: Pacemaker's policy engine did not properly handle constraints on a remote node if the remote node's remote connection was on a failed cluster node and unrecoverable. Consequence: Pacemaker could not calculate what actions to take if a remote connection host needed to be fenced, and the remote connection could not be migrated elsewhere, and certain constraints were in effect. Fix: Constraints are now handled properly. Result: Pacemaker now successfully calculates what actions to take in that situation.
Clone Of:
Environment:
Last Closed: 2018-04-10 15:28:37 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
crm report (239.26 KB, application/x-bzip)
2016-04-04 02:27 UTC, Andrew Beekhof 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:0860 0 None None None 2018-04-10 15:29:53 UTC

Description Andrew Beekhof 2016-04-04 02:27:49 UTC 
Created attachment 1143148 [details]
crm report

Description of problem:


[root@airfrance-2 ~]# crm_simulate -SL

Current cluster status:
Node airfrance-1 (1): UNCLEAN (offline)
Online: [ airfrance-2 ]
RemoteOnline: [ airfrance-3 ]

 airfrance-3	(ocf::pacemaker:remote):	Started airfrance-1 (UNCLEAN)
 killer	(stonith:fence_xvm):	Started airfrance-2
 test	(ocf::heartbeat:Dummy):	Started airfrance-3
 Master/Slave Set: galera-master [galera]
     galera	(ocf::pacemaker:Stateful):	Master airfrance-1 (UNCLEAN)
     Masters: [ airfrance-2 ]
     Stopped: [ airfrance-3 ]

Transition Summary:
 * Stop    airfrance-3	(airfrance-1)
 * Stop    test	(airfrance-3 - blocked)
 * Demote  galera:0	(Master -> Stopped airfrance-1)

Executing cluster transition:
 * Fencing airfrance-1 (reboot)
 * Pseudo action:   stonith_complete
 * Pseudo action:   airfrance-3_stop_0
Transition failed: terminated
An invalid transition was produced

[...]

[root@airfrance-2 ~]# pcs resource cleanup test
Waiting for 1 replies from the CRMd. OK
Cleaning up test on airfrance-2, removing fail-count-test
Cleaning up test on airfrance-3, removing fail-count-test


[root@airfrance-2 ~]# crm_simulate -SL

Current cluster status:
Node airfrance-1 (1): UNCLEAN (offline)
Online: [ airfrance-2 ]
RemoteOnline: [ airfrance-3 ]

 airfrance-3	(ocf::pacemaker:remote):	Started airfrance-1 (UNCLEAN)
 killer	(stonith:fence_xvm):	Started airfrance-2
 test	(ocf::heartbeat:Dummy):	Started airfrance-3
 Master/Slave Set: galera-master [galera]
     galera	(ocf::pacemaker:Stateful):	Master airfrance-1 (UNCLEAN)
     Masters: [ airfrance-2 ]
     Stopped: [ airfrance-3 ]

Transition Summary:
 * Stop    airfrance-3	(airfrance-1)
 * Stop    test	(airfrance-3 - blocked)
 * Demote  galera:0	(Master -> Stopped airfrance-1)

Executing cluster transition:
 * Fencing airfrance-1 (reboot)
 * Pseudo action:   stonith_complete
 * Pseudo action:   airfrance-3_stop_0
Transition failed: terminated
An invalid transition was produced

[...]

[root@airfrance-2 ~]# pcs resource cleanup airfrance-3
Waiting for 1 replies from the CRMd. OK
Cleaning up airfrance-3 on airfrance-2, removing fail-count-airfrance-3
Cleaning up airfrance-3 on airfrance-3, removing fail-count-airfrance-3

[root@airfrance-2 ~]# crm_simulate -SL

Current cluster status:
Node airfrance-1 (1): UNCLEAN (offline)
Online: [ airfrance-2 ]
RemoteOnline: [ airfrance-3 ]

 airfrance-3	(ocf::pacemaker:remote):	Started airfrance-1 (UNCLEAN)
 killer	(stonith:fence_xvm):	Started airfrance-2
 test	(ocf::heartbeat:Dummy):	Started airfrance-3
 Master/Slave Set: galera-master [galera]
     galera	(ocf::pacemaker:Stateful):	Master airfrance-1 (UNCLEAN)
     Masters: [ airfrance-2 ]
     Stopped: [ airfrance-3 ]

Transition Summary:
 * Stop    airfrance-3	(airfrance-1)
 * Stop    test	(airfrance-3 - blocked)
 * Demote  galera:0	(Master -> Stopped airfrance-1)

Executing cluster transition:
 * Fencing airfrance-1 (reboot)
 * Pseudo action:   stonith_complete
 * Pseudo action:   airfrance-3_stop_0
Transition failed: terminated
An invalid transition was produced

[...]

[root@airfrance-2 ~]# pcs stonith confirm airfrance-0
Node: airfrance-0 confirmed fenced
[root@airfrance-2 ~]# crm_simulate -SL

Current cluster status:
Node airfrance-1 (1): UNCLEAN (offline)
Online: [ airfrance-2 ]
RemoteOnline: [ airfrance-3 ]

 airfrance-3	(ocf::pacemaker:remote):	Started airfrance-1 (UNCLEAN)
 killer	(stonith:fence_xvm):	Started airfrance-2
 test	(ocf::heartbeat:Dummy):	Started airfrance-3
 Master/Slave Set: galera-master [galera]
     galera	(ocf::pacemaker:Stateful):	Master airfrance-1 (UNCLEAN)
     Masters: [ airfrance-2 ]
     Stopped: [ airfrance-3 ]

Transition Summary:
 * Stop    airfrance-3	(airfrance-1)
 * Stop    test	(airfrance-3 - blocked)
 * Demote  galera:0	(Master -> Stopped airfrance-1)

Executing cluster transition:
 * Fencing airfrance-1 (reboot)
 * Pseudo action:   stonith_complete
 * Pseudo action:   airfrance-3_stop_0
Transition failed: terminated
An invalid transition was produced

[...]

Version-Release number of selected component (if applicable):

Pacemaker 1.1.13-10.el7_2.2


How reproducible:

trivial

Steps to Reproduce:
1. 2 nodes + 1 remote
2. constraints as per crm report
3. virsh destroy node 1

Actual results:

cluster fully locked up because the stop action for the resource inside the remote node cannot run

- cleanup of the resource in the remote node: no effect
- cleanup of the remote node resource: no effect
- confirm stonith action: no effect 

Expected results:

even if the transition cant complete (which it should!) at least one of the cleanup actions should unblock the cluster

Additional info:

see attachment (if crm_report ever completes due to bug #1323544)
Comment 2 Andrew Beekhof 2016-04-05 02:35:12 UTC 
A combination of factors were at play here:

1. Fencing was temporarily busted (fence_xvm returned no hosts)

2. If you look at the transcript above, I confirmed "airfrance-0" which doesn't exist instead of airfrance-1

3. The constraint:
      <rsc_location id="cli-ban-airfrance-3-on-airfrance-2" rsc="airfrance-3" role="Started" node="airfrance-2" score="-INFINITY"/>

   prevented the connection resource from being started on the remaining node, this tripped up the "i don't really need to stop resources on the remote node" logic.

Which needs to be fixed but can be done at a lower priority.
Comment 3 Ken Gaillot 2016-07-05 20:59:56 UTC 
Capacity constrained, moving to 7.4
Comment 4 Ken Gaillot 2017-01-16 17:13:41 UTC 
Unfortunately this is unlikely to be addressed in 7.4 timeframe
Comment 6 Ken Gaillot 2017-10-13 19:55:52 UTC 
The invalid transition can be reproduced with upstream versions through 1.1.16, but no longer occurs with upstream 1.1.17 and later, so it was fixed somewhere along the way. I'll add an anonymized regression test for it.

QA: To test, grab cib.xml.live from the attached crm_report and run "crm_simulate -Sx cib.xml.live". Before the fix, it outputs "Invalid transition", while after the fix, the simulation completes successfully.
Comment 11 errata-xmlrpc 2018-04-10 15:28:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0860
Note You need to log in before you can comment on or make changes to this bug.