RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1712584 - Option to skip resource recovery after clean node shutdown (Zero Touch)
Summary: Option to skip resource recovery after clean node shutdown (Zero Touch)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: pacemaker
Version: 8.1
Hardware: Unspecified
OS: Unspecified
urgent
medium
Target Milestone: pre-dev-freeze
: 8.2
Assignee: Ken Gaillot
QA Contact: Markéta Smazová
Steven J. Levine
URL:
Whiteboard:
Depends On:
Blocks: 1781820
TreeView+ depends on / blocked
 
Reported: 2019-05-21 20:39 UTC by Ken Gaillot
Modified: 2023-12-15 16:30 UTC (History)
6 users (show)

Fixed In Version: pacemaker-2.0.3-4.el8
Doc Type: Enhancement
Doc Text:
.Pacemaker support for configuring resources to remain stopped on clean node shutdown When a cluster node shuts down, Pacemaker’s default response is to stop all resources running on that node and recover them elsewhere. Some users prefer to have high availability only for failures, and to treat clean shutdowns as scheduled outages. To address this, Pacemaker now supports the `shutdown-lock` and `shutdown-lock-limit` cluster properties to specify that resources active on a node when it shuts down should remain stopped until the node next rejoins. Users can now use clean shutdowns as scheduled outages without any manual intervention. For information on configuring resources to remain stopped on a clean node shutdown, see link: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_and_managing_high_availability_clusters/index#assembly_configuring-resources-to-remain-stopped-configuring-and-managing-high-availability-clusters[Configuring resources to remain stopped on clean node shutdown].
Clone Of:
: 1781820 (view as bug list)
Environment:
Last Closed: 2020-04-28 15:38:28 UTC
Type: Feature Request
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 5225251 0 None None None 2020-09-21 21:17:43 UTC
Red Hat Knowledge Base (Solution) 5227211 0 None None None 2020-09-21 21:18:46 UTC
Red Hat Product Errata RHEA-2020:1609 0 None None None 2020-04-28 15:39:10 UTC

Internal Links: 1642140

Description Ken Gaillot 2019-05-21 20:39:54 UTC
Pacemaker's current response to a clean node shutdown is to stop all resources running on it and allow them to be recovered elsewhere (i.e. high availability).

However some users would like the option to leave the node's resources stopped in this situation (while still recovering resources for failures).

Comment 1 Ken Gaillot 2019-05-21 20:52:11 UTC
Possible implementation:

A new resource meta-attribute "shutdown-recovery" would default to "true" to keep the current behavior. It could be set in rsc_defaults to apply cluster-wide.

When a clean node is scheduled for shutdown, the scheduler would add a list of resources running on the node with shutdown-recovery=false to the shutdown graph action.

When the DC controller processes the shutdown graph action, it would append that list to a new CIB section within node_state. (Appending would handle the case when a new transition is scheduled when some of the resources have already been stopped.)

When the scheduler sees the new section present, it would ban those resources from all other nodes, as long as the original node is clean.

When a node becomes a member, the DC controller would erase the new section.

Special consideration would need to be given to clones (anonymous, unique, and promotable), maintenance mode (cluster-wide and per-node), unmanaged resources, shutdown failures, remote and guest nodes, bundles, the possibility of race conditions, and the ability to manually clear the new CIB section.

Comment 2 Andrew Beekhof 2019-05-21 21:44:51 UTC
While technically possible to implement, this seems like a terrible idea that could also be achieved through a liberal use of location constraints. I’d predict such a feature to open a can of worms if implemented

Comment 4 Ken Gaillot 2019-05-21 22:00:34 UTC
Another possible implementation:

We could add a new option to crm_resource (e.g. "--lock-resources"?) that would run the scheduler to get a list of resources running on the node at the moment, and add -INFINITY constraints for those resources on all other nodes. crm_resource --clear could get rid of the constraints.

That would require being able to run the relevant command at node start-up and shutdown. This might be doable via systemd drop-in units.

Comment 5 Andrew Beekhof 2019-05-22 20:38:21 UTC
That does sound less worse :)

Comment 8 Ken Gaillot 2019-12-20 00:54:52 UTC
After further investigation, I rejected the idea of a new crm_resource option invoked by systemd unit overrides. The main downside is that the cluster can be stopped by means other than systemd (e.g. sending SIGTERM to pacemakerd, setting a node's shutdown node attribute, or running crmadmin -K -- whether by the user or a higher-level tool). This approach would also have some problematic corner cases (e.g. if the unlock command failed or timed out after cluster start, the resources would remain locked without any indication to the user), and it would not be usable on remote nodes.

Another possibility I considered and rejected was a new transient_constraints section in the node's node_state in the CIB. When stopping resources on a node that's shutting down (and a new cluster property is set), the scheduler would create a new graph op with a list of constraints that need to be added. The controller would add the constraints to the new section before doing any of the stops. The transient constraint section would be cleared when the node rejoins. The main downside here is that location preferences would be required to ensure that the stopped resources start on the same node. It also had some corner cases.

I ended up going with two new cluster properties shutdown-lock and shutdown-lock-expiration. If shutdown-lock is set to the default of false, the cluster will recover resources that are active on nodes being cleanly shut down (i.e. the current behavior). If shutdown-lock is true, such resources will be "locked" to the node, unable to start elsewhere, until they start on the node again after it rejoins. If shutdown-lock-expiration is set (as a time duration other than the default of 0), resources will be unlocked if the node does not rejoin within this much time since its shutdown was initiated.

This is implemented using a new "shutdown-lock" XML attribute on the lrm_resource element in resource operation history. When the resource is stopping on a node that's cleanly shutting down, this lock will be recorded with the stop operation (if the stop is successful). When the resource starts again after the node rejoins, the lock will be cleared with the start operation (successful or not). The scheduler bans the resource from all other nodes while the lock is in effect.

Benefits of this approach: no timing issues are possible, since the lock is recorded and cleared simultaneously with the relevant operation results; it works whether or not a DC has been elected at any given point in the process; it works regardless of how the cluster is started or stopped; it works for remote nodes; and fencing the node, or any of the resource actions failing, will clear the locks automatically.

Regardless of implementation, it is important to note that once resources are unlocked, the cluster by definition is free to move them elsewhere, and it might do so. Stickiness or location preferences can be used to make this less likely.

Comment 10 Ken Gaillot 2020-01-16 20:27:44 UTC
This has been implemented upstream as of https://github.com/ClusterLabs/pacemaker/pull/1974

The only difference from what is described in Comment 8 is that the proposed shutdown-lock-expiration option was renamed to shutdown-lock-limit.

Locks can be manually cleared using "pcs resource refresh <resource> --node <node>". Both resource and node must be specified.

Shutdown locks work with remote nodes as well as cluster nodes, but lock expiration and manual clearing work only if the remote node's connection resource was disabled. If a remote node is shut down without disabling the resource, the lock will remain in effect until the remote node is brought back up regardless of any shutdown-lock-limit or manual refresh.

Comment 11 Ken Gaillot 2020-01-17 02:35:40 UTC
QE: To test:

1. Configure a cluster of at least two nodes (so you can shut down one and retain quorum) with at least one resource.
2. If shutdown-lock is not specified or specified as false, shutting down a node running a resource will result in the resource being recovered on another node.
3. If shutdown-lock is specified as true, shutting down a node running a resource will result in the resource being stopped.
- "pcs status" will show the resource as "LOCKED" while the node is down.
- Starting the node again will result in the resource starting there again. (If the cluster has multiple resources, load balancing could result in it moving afterward, unless stickiness or a location preference is used.)
- Running "pcs resource refresh <resource> --node <node>" while the node is down and the resource is locked will result in the lock being removed, and the resource starting elsewhere.
- If shutdown-lock-limit is specified as an amount of time (e.g. 5min), then after that much time has passed since the shutdown was initiated (within the granularity of the cluster-recheck-interval), the resource will start elsewhere even if the node remains down.

shutdown-lock will work for either cluster nodes or remote nodes, but not guest nodes. shutdown-lock-limit and manual refresh will work with remote nodes only if the remote node is stopped via "systemctl stop pacemaker_remote" on the remote node, followed by "pcs resource disable <remote-connection-resource>", followed by the lock expiration or refresh.

Comment 12 Ken Gaillot 2020-01-17 02:36:54 UTC
QE: clarification: the cluster-recheck-interval does not matter in 8.2, that comment was for 7.8, sorry. In 8.2, the recheck interval will automatically be adjusted to account for things such as shutdown lock expiration and failure timeouts.

Comment 15 Markéta Smazová 2020-02-07 13:58:46 UTC
CASE 1
-------
Property "shutdown-lock" is not specified or is specified as false. Shutting down a node running a resource will result in the resource being recovered on another node (current behavior).


>   [root@virt-158 ~]# rpm -q pacemaker
>   pacemaker-2.0.3-4.el8.x86_64

Verify whether "shutdown-lock" property is false.

>   [root@virt-158 ~]# pcs property list --all | grep shutdown-lock
>    shutdown-lock: false
>    shutdown-lock-limit: 0

There are two resources "first" and "fourth" on virt-156.

>   [root@virt-158 ~]# pcs status
>   Cluster name: STSRHTS26675
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum
>     * Last updated: Wed Jan 29 18:01:16 2020
>     * Last change:  Wed Jan 29 18:01:10 2020 by root via cibadmin on virt-158
>     * 3 nodes configured
>     * 13 resource instances configured

>   Node List:
>     * Online: [ virt-156 virt-157 virt-158 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-156
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-158
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-156 virt-157 virt-158 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-156
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-158
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-156

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Shutting down virt-156.

>   [root@virt-158 ~]# qarsh virt-156 pcs cluster stop
>   Stopping Cluster (pacemaker)...
>   Stopping Cluster (corosync)...

Node virt-156 is offline, resources "first" and "fourth" are recovered on other nodes.

>   [root@virt-158 ~]# pcs status
>   Cluster name: STSRHTS26675
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum
>     * Last updated: Wed Jan 29 18:12:39 2020
>     * Last change:  Wed Jan 29 18:01:10 2020 by root via cibadmin on virt-158
>     * 3 nodes configured
>     * 13 resource instances configured

>   Node List:
>     * Online: [ virt-157 virt-158 ]
>     * OFFLINE: [ virt-156 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-158
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-157 virt-158 ]
>       * Stopped: [ virt-156 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-157
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-158
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-157

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Start cluster on the node again so that it rejoins the cluster.

>   [root@virt-158 ~]# qarsh virt-156 pcs cluster start
>   Starting Cluster...

>   [root@virt-158 ~]# pcs status
>   Cluster name: STSRHTS26675
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum
>     * Last updated: Wed Jan 29 18:31:09 2020
>     * Last change:  Wed Jan 29 18:17:25 2020 by hacluster via crmd on virt-158
>     * 3 nodes configured
>     * 13 resource instances configured

>   Node List:
>     * Online: [ virt-156 virt-157 virt-158 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-156
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-156 virt-157 virt-158 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-157
>     * second	(ocf::pacemaker:Dummy):	Started virt-156
>     * third	(ocf::pacemaker:Dummy):	Started virt-158
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-157

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled


CASE 2
-------
Property "shutdown-lock" is specified as true, shutting down a node running a resource will result in the resource being stopped. "pcs status" will show the resource as "LOCKED" while the node is down. Starting the node again will result in the resource starting there again.

Set "shutdown-lock" to true and verify.

>   [root@virt-158 ~]# pcs property set shutdown-lock=true
>   [root@virt-158 ~]# pcs property list --all | grep shutdown-lock
>    shutdown-lock: true
>    shutdown-lock-limit: 0

There are resources "third" and "fifth" on virt-156.

>   [root@virt-158 ~]# pcs status
>   [...output omitted...]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-156
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-156 virt-157 virt-158 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-156
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-157
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-156
>
>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Shutting down virt-156.

>   [root@virt-158 ~]# qarsh virt-156 pcs cluster stop
>   Stopping Cluster (pacemaker)...
>   Stopping Cluster (corosync)...

Node virt-156 is offline, resources "third" and "fifth" are marked as (LOCKED).

>   [root@virt-158 ~]# pcs status
>   [...output omitted...]

>   Node List:
>     * Online: [ virt-157 virt-158 ]
>     * OFFLINE: [ virt-156 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-157
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-157 virt-158 ]
>       * Stopped: [ virt-156 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Stopped virt-156 (LOCKED)
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-158
>     * fifth	(ocf::pacemaker:Dummy):	Stopped virt-156 (LOCKED)

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Start cluster on the node again so that it rejoins the cluster. Locked resources should get started, not necessarily on the same node (due to eg. load balancing).

>   [root@virt-158 ~]# qarsh virt-156 pcs cluster start
>   Starting Cluster...

Resources "third" and "fifth" are recovered on node virt-156.

>   [root@virt-158 ~]# pcs status
>   [...output omitted...]

>   Node List:
>     * Online: [ virt-156 virt-157 virt-158 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-156
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-156 virt-157 virt-158 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-156
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-158
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-156

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled


CASE 3
-------
Property "shutdown-lock" is specified as true. Shutting down a node and running "pcs resource refresh <resource> node=<node>" (both resource and node must be specified) while the node is down and the resource is locked, will result in the lock being removed, and the resource starting elsewhere. 

Verify whether "shutdown-lock" property is true.

>   [root@virt-158 ~]# pcs property list --all | grep shutdown-lock
>    shutdown-lock: true
>    shutdown-lock-limit: 0
>
>   [root@virt-158 ~]# pcs status
>   [...output omitted...]

>   Node List:
>     * Online: [ virt-156 virt-157 virt-158 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-156
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-156 virt-157 virt-158 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-156
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-158
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-156

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Shutting down virt-156.

>   [root@virt-158 ~]# qarsh virt-156 pcs cluster stop
>   Stopping Cluster (pacemaker)...
>   Stopping Cluster (corosync)...

Node virt-156 is offline, resources "third" and "fifth" are marked as (LOCKED).

>   [root@virt-158 ~]# pcs status
>   [...output omitted...]

>   Node List:
>     * Online: [ virt-157 virt-158 ]
>     * OFFLINE: [ virt-156 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-157
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-157 virt-158 ]
>       * Stopped: [ virt-156 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Stopped virt-156 (LOCKED)
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-158
>     * fifth	(ocf::pacemaker:Dummy):	Stopped virt-156 (LOCKED)

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Refreshing resource "third" and checking the status. Node virt-156 is still offline, resource "third" restarted on virt-158.

>   [root@virt-158 ~]# pcs resource refresh third node=virt-156
>   Cleaned up third on virt-156

>     * 'third' is locked to node virt-156 due to shutdown
>   Waiting for 1 reply from the controller. OK

>   [root@virt-158 ~]# pcs status
>   [...output omitted...]

>   Node List:
>     * Online: [ virt-157 virt-158 ]
>     * OFFLINE: [ virt-156 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-157
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-157 virt-158 ]
>       * Stopped: [ virt-156 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-158
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-158
>     * fifth	(ocf::pacemaker:Dummy):	Stopped virt-156 (LOCKED)

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Refreshing resource "fifth" and checking the status. Node virt-156 is still offline, resource "fifth" restarted on virt-157.

>   [root@virt-158 ~]# pcs resource refresh fifth node=virt-156
>   Cleaned up fifth on virt-156

>     * 'fifth' is locked to node virt-156 due to shutdown
>   Waiting for 1 reply from the controller. OK

>   [root@virt-158 ~]# pcs status
>   [...output omitted...]

>   Node List:
>     * Online: [ virt-157 virt-158 ]
>     * OFFLINE: [ virt-156 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-157
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-157 virt-158 ]
>       * Stopped: [ virt-156 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-158
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-158
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-157

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Start cluster on the node again so that it rejoins the cluster.

>   [root@virt-158 ~]# qarsh virt-156 pcs cluster start
>   Starting Cluster...

>   [root@virt-158 ~]# pcs status
>   [...output omitted...]

>   Node List:
>     * Online: [ virt-156 virt-157 virt-158 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-156
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-156 virt-157 virt-158 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-156
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-158
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-157

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled


CASE 4
-------
Property "shutdown-lock" is specified as true and "shutdown-lock-limit" is specified as an amount of time (5 min) and the node shutdown is initiated. After the "shutdown-lock-limit" expires, locked resources will get unlocked and started elsewhere, while the node remains down.
In 7.8 this was also dependent on "cluster-recheck-interval" setting, in 8.2 the "cluster-recheck-interval" is automatically adjusted to account for "shutdown-lock" expiration and failure timeouts. See bz#1747553 for more details.

Verify whether "shutdown-lock" is set to true and check the setting of "shutdown-lock-limit" property.

>   [root@virt-158 16:27:37 ~]# pcs property list --all | grep shutdown-lock
>    shutdown-lock: true
>    shutdown-lock-limit: 0

Set "shutdown-lock-limit" to 5 minutes and verify.

>   [root@virt-158 16:27:47 ~]# pcs property set shutdown-lock-limit=5min
>   [root@virt-158 16:27:55 ~]# pcs property list --all | grep shutdown-lock
>    shutdown-lock: true
>    shutdown-lock-limit: 5min

>   [root@virt-158 16:28:09 ~]# pcs status
>   Cluster name: STSRHTS26675
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum
>     * Last updated: Thu Jan 30 16:28:16 2020
>     * Last change:  Thu Jan 30 16:27:55 2020 by root via cibadmin on virt-158
>     * 3 nodes configured
>     * 15 resource instances configured

>   Node List:
>     * Online: [ virt-156 virt-157 virt-158 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-156
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-156 virt-157 virt-158 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-156
>     * second	(ocf::pacemaker:Dummy):	Started virt-157
>     * third	(ocf::pacemaker:Dummy):	Started virt-158
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-156
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-157
>     * sixth	(ocf::pacemaker:Dummy):	Started virt-158

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Shutting down node virt-156.

>   [root@virt-158 16:28:17 ~]# qarsh virt-156 pcs cluster stop
>   Stopping Cluster (pacemaker)...
>   Stopping Cluster (corosync)...

Resources "first" and "fourth" on virt-156 are marked as (LOCKED), node virt-156 remains down.

>   [root@virt-158 16:28:29 ~]# pcs status
>   [...output omitted...]

>   Node List:
>     * Online: [ virt-157 virt-158 ]
>     * OFFLINE: [ virt-156 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-157
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-157 virt-158 ]
>       * Stopped: [ virt-156 ]
>     * first	(ocf::pacemaker:Dummy):	Stopped virt-156 (LOCKED)
>     * second	(ocf::pacemaker:Dummy):	Started virt-158
>     * third	(ocf::pacemaker:Dummy):	Started virt-158
>     * fourth	(ocf::pacemaker:Dummy):	Stopped virt-156 (LOCKED)
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-157
>     * sixth	(ocf::pacemaker:Dummy):	Started virt-158

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Waiting for 5 minutes. The resources have started on other nodes and virt-156 is still offline.

>   [root@virt-158 16:34:44 ~]# pcs status
>   Cluster name: STSRHTS26675
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum
>     * Last updated: Thu Jan 30 16:34:48 2020
>     * Last change:  Thu Jan 30 16:27:55 2020 by root via cibadmin on virt-158
>     * 3 nodes configured
>     * 15 resource instances configured

>   Node List:
>     * Online: [ virt-157 virt-158 ]
>     * OFFLINE: [ virt-156 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-157
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-157 virt-158 ]
>       * Stopped: [ virt-156 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-158
>     * third	(ocf::pacemaker:Dummy):	Started virt-157
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-157
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-158
>     * sixth	(ocf::pacemaker:Dummy):	Started virt-158

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

Start cluster on the node again so that it rejoins the cluster.

>   [root@virt-158 16:34:48 ~]# qarsh virt-156 pcs cluster start
>   Starting Cluster...

>   [root@virt-158 16:37:00 ~]# pcs status
>   Cluster name: STSRHTS26675
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum
>     * Last updated: Thu Jan 30 16:37:10 2020
>     * Last change:  Thu Jan 30 16:27:55 2020 by root via cibadmin on virt-158
>     * 3 nodes configured
>     * 15 resource instances configured

>   Node List:
>     * Online: [ virt-156 virt-157 virt-158 ]

>   Full List of Resources:
>     * fence-virt-156	(stonith:fence_xvm):	Started virt-157
>     * fence-virt-157	(stonith:fence_xvm):	Started virt-158
>     * fence-virt-158	(stonith:fence_xvm):	Started virt-156
>     * Clone Set: locking-clone [locking]:
>       * Started: [ virt-156 virt-157 virt-158 ]
>     * first	(ocf::pacemaker:Dummy):	Started virt-158
>     * second	(ocf::pacemaker:Dummy):	Started virt-156
>     * third	(ocf::pacemaker:Dummy):	Started virt-157
>     * fourth	(ocf::pacemaker:Dummy):	Started virt-157
>     * fifth	(ocf::pacemaker:Dummy):	Started virt-158
>     * sixth	(ocf::pacemaker:Dummy):	Started virt-156

>   Daemon Status:
>     corosync: active/enabled
>     pacemaker: active/enabled
>     pcsd: active/enabled

marking verified in pacemaker-2.0.3-4.el8

Comment 17 errata-xmlrpc 2020-04-28 15:38:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1609


Note You need to log in before you can comment on or make changes to this bug.