Bug 1712584
| Summary: | Option to skip resource recovery after clean node shutdown (Zero Touch) | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Ken Gaillot <kgaillot> | |
| Component: | pacemaker | Assignee: | Ken Gaillot <kgaillot> | |
| Status: | CLOSED ERRATA | QA Contact: | Markéta Smazová <msmazova> | |
| Severity: | medium | Docs Contact: | Steven J. Levine <slevine> | |
| Priority: | urgent | |||
| Version: | 8.1 | CC: | cfeist, cluster-maint, djansa, mnovacek, sbradley, slevine | |
| Target Milestone: | pre-dev-freeze | |||
| Target Release: | 8.2 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | pacemaker-2.0.3-4.el8 | Doc Type: | Enhancement | |
| Doc Text: |
.Pacemaker support for configuring resources to remain stopped on clean node shutdown
When a cluster node shuts down, Pacemaker’s default response is to stop all resources running on that node and recover them elsewhere. Some users prefer to have high availability only for failures, and to treat clean shutdowns as scheduled outages. To address this, Pacemaker now supports the `shutdown-lock` and `shutdown-lock-limit` cluster properties to specify that resources active on a node when it shuts down should remain stopped until the node next rejoins. Users can now use clean shutdowns as scheduled outages without any manual intervention. For information on configuring resources to remain stopped on a clean node shutdown, see link: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_and_managing_high_availability_clusters/index#assembly_configuring-resources-to-remain-stopped-configuring-and-managing-high-availability-clusters[Configuring resources to remain stopped on clean node shutdown].
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1781820 (view as bug list) | Environment: | ||
| Last Closed: | 2020-04-28 15:38:28 UTC | Type: | Feature Request | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1781820 | |||
|
Description
Ken Gaillot
2019-05-21 20:39:54 UTC
Possible implementation: A new resource meta-attribute "shutdown-recovery" would default to "true" to keep the current behavior. It could be set in rsc_defaults to apply cluster-wide. When a clean node is scheduled for shutdown, the scheduler would add a list of resources running on the node with shutdown-recovery=false to the shutdown graph action. When the DC controller processes the shutdown graph action, it would append that list to a new CIB section within node_state. (Appending would handle the case when a new transition is scheduled when some of the resources have already been stopped.) When the scheduler sees the new section present, it would ban those resources from all other nodes, as long as the original node is clean. When a node becomes a member, the DC controller would erase the new section. Special consideration would need to be given to clones (anonymous, unique, and promotable), maintenance mode (cluster-wide and per-node), unmanaged resources, shutdown failures, remote and guest nodes, bundles, the possibility of race conditions, and the ability to manually clear the new CIB section. While technically possible to implement, this seems like a terrible idea that could also be achieved through a liberal use of location constraints. I’d predict such a feature to open a can of worms if implemented Another possible implementation: We could add a new option to crm_resource (e.g. "--lock-resources"?) that would run the scheduler to get a list of resources running on the node at the moment, and add -INFINITY constraints for those resources on all other nodes. crm_resource --clear could get rid of the constraints. That would require being able to run the relevant command at node start-up and shutdown. This might be doable via systemd drop-in units. That does sound less worse :) After further investigation, I rejected the idea of a new crm_resource option invoked by systemd unit overrides. The main downside is that the cluster can be stopped by means other than systemd (e.g. sending SIGTERM to pacemakerd, setting a node's shutdown node attribute, or running crmadmin -K -- whether by the user or a higher-level tool). This approach would also have some problematic corner cases (e.g. if the unlock command failed or timed out after cluster start, the resources would remain locked without any indication to the user), and it would not be usable on remote nodes. Another possibility I considered and rejected was a new transient_constraints section in the node's node_state in the CIB. When stopping resources on a node that's shutting down (and a new cluster property is set), the scheduler would create a new graph op with a list of constraints that need to be added. The controller would add the constraints to the new section before doing any of the stops. The transient constraint section would be cleared when the node rejoins. The main downside here is that location preferences would be required to ensure that the stopped resources start on the same node. It also had some corner cases. I ended up going with two new cluster properties shutdown-lock and shutdown-lock-expiration. If shutdown-lock is set to the default of false, the cluster will recover resources that are active on nodes being cleanly shut down (i.e. the current behavior). If shutdown-lock is true, such resources will be "locked" to the node, unable to start elsewhere, until they start on the node again after it rejoins. If shutdown-lock-expiration is set (as a time duration other than the default of 0), resources will be unlocked if the node does not rejoin within this much time since its shutdown was initiated. This is implemented using a new "shutdown-lock" XML attribute on the lrm_resource element in resource operation history. When the resource is stopping on a node that's cleanly shutting down, this lock will be recorded with the stop operation (if the stop is successful). When the resource starts again after the node rejoins, the lock will be cleared with the start operation (successful or not). The scheduler bans the resource from all other nodes while the lock is in effect. Benefits of this approach: no timing issues are possible, since the lock is recorded and cleared simultaneously with the relevant operation results; it works whether or not a DC has been elected at any given point in the process; it works regardless of how the cluster is started or stopped; it works for remote nodes; and fencing the node, or any of the resource actions failing, will clear the locks automatically. Regardless of implementation, it is important to note that once resources are unlocked, the cluster by definition is free to move them elsewhere, and it might do so. Stickiness or location preferences can be used to make this less likely. This has been implemented upstream as of https://github.com/ClusterLabs/pacemaker/pull/1974 The only difference from what is described in Comment 8 is that the proposed shutdown-lock-expiration option was renamed to shutdown-lock-limit. Locks can be manually cleared using "pcs resource refresh <resource> --node <node>". Both resource and node must be specified. Shutdown locks work with remote nodes as well as cluster nodes, but lock expiration and manual clearing work only if the remote node's connection resource was disabled. If a remote node is shut down without disabling the resource, the lock will remain in effect until the remote node is brought back up regardless of any shutdown-lock-limit or manual refresh. QE: To test: 1. Configure a cluster of at least two nodes (so you can shut down one and retain quorum) with at least one resource. 2. If shutdown-lock is not specified or specified as false, shutting down a node running a resource will result in the resource being recovered on another node. 3. If shutdown-lock is specified as true, shutting down a node running a resource will result in the resource being stopped. - "pcs status" will show the resource as "LOCKED" while the node is down. - Starting the node again will result in the resource starting there again. (If the cluster has multiple resources, load balancing could result in it moving afterward, unless stickiness or a location preference is used.) - Running "pcs resource refresh <resource> --node <node>" while the node is down and the resource is locked will result in the lock being removed, and the resource starting elsewhere. - If shutdown-lock-limit is specified as an amount of time (e.g. 5min), then after that much time has passed since the shutdown was initiated (within the granularity of the cluster-recheck-interval), the resource will start elsewhere even if the node remains down. shutdown-lock will work for either cluster nodes or remote nodes, but not guest nodes. shutdown-lock-limit and manual refresh will work with remote nodes only if the remote node is stopped via "systemctl stop pacemaker_remote" on the remote node, followed by "pcs resource disable <remote-connection-resource>", followed by the lock expiration or refresh. QE: clarification: the cluster-recheck-interval does not matter in 8.2, that comment was for 7.8, sorry. In 8.2, the recheck interval will automatically be adjusted to account for things such as shutdown lock expiration and failure timeouts. CASE 1 ------- Property "shutdown-lock" is not specified or is specified as false. Shutting down a node running a resource will result in the resource being recovered on another node (current behavior). > [root@virt-158 ~]# rpm -q pacemaker > pacemaker-2.0.3-4.el8.x86_64 Verify whether "shutdown-lock" property is false. > [root@virt-158 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: false > shutdown-lock-limit: 0 There are two resources "first" and "fourth" on virt-156. > [root@virt-158 ~]# pcs status > Cluster name: STSRHTS26675 > Cluster Summary: > * Stack: corosync > * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum > * Last updated: Wed Jan 29 18:01:16 2020 > * Last change: Wed Jan 29 18:01:10 2020 by root via cibadmin on virt-158 > * 3 nodes configured > * 13 resource instances configured > Node List: > * Online: [ virt-156 virt-157 virt-158 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-156 > * fence-virt-157 (stonith:fence_xvm): Started virt-157 > * fence-virt-158 (stonith:fence_xvm): Started virt-158 > * Clone Set: locking-clone [locking]: > * Started: [ virt-156 virt-157 virt-158 ] > * first (ocf::pacemaker:Dummy): Started virt-156 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-158 > * fourth (ocf::pacemaker:Dummy): Started virt-156 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Shutting down virt-156. > [root@virt-158 ~]# qarsh virt-156 pcs cluster stop > Stopping Cluster (pacemaker)... > Stopping Cluster (corosync)... Node virt-156 is offline, resources "first" and "fourth" are recovered on other nodes. > [root@virt-158 ~]# pcs status > Cluster name: STSRHTS26675 > Cluster Summary: > * Stack: corosync > * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum > * Last updated: Wed Jan 29 18:12:39 2020 > * Last change: Wed Jan 29 18:01:10 2020 by root via cibadmin on virt-158 > * 3 nodes configured > * 13 resource instances configured > Node List: > * Online: [ virt-157 virt-158 ] > * OFFLINE: [ virt-156 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-158 > * Clone Set: locking-clone [locking]: > * Started: [ virt-157 virt-158 ] > * Stopped: [ virt-156 ] > * first (ocf::pacemaker:Dummy): Started virt-157 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-158 > * fourth (ocf::pacemaker:Dummy): Started virt-157 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Start cluster on the node again so that it rejoins the cluster. > [root@virt-158 ~]# qarsh virt-156 pcs cluster start > Starting Cluster... > [root@virt-158 ~]# pcs status > Cluster name: STSRHTS26675 > Cluster Summary: > * Stack: corosync > * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum > * Last updated: Wed Jan 29 18:31:09 2020 > * Last change: Wed Jan 29 18:17:25 2020 by hacluster via crmd on virt-158 > * 3 nodes configured > * 13 resource instances configured > Node List: > * Online: [ virt-156 virt-157 virt-158 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-156 > * Clone Set: locking-clone [locking]: > * Started: [ virt-156 virt-157 virt-158 ] > * first (ocf::pacemaker:Dummy): Started virt-157 > * second (ocf::pacemaker:Dummy): Started virt-156 > * third (ocf::pacemaker:Dummy): Started virt-158 > * fourth (ocf::pacemaker:Dummy): Started virt-157 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled CASE 2 ------- Property "shutdown-lock" is specified as true, shutting down a node running a resource will result in the resource being stopped. "pcs status" will show the resource as "LOCKED" while the node is down. Starting the node again will result in the resource starting there again. Set "shutdown-lock" to true and verify. > [root@virt-158 ~]# pcs property set shutdown-lock=true > [root@virt-158 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: true > shutdown-lock-limit: 0 There are resources "third" and "fifth" on virt-156. > [root@virt-158 ~]# pcs status > [...output omitted...] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-156 > * Clone Set: locking-clone [locking]: > * Started: [ virt-156 virt-157 virt-158 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-156 > * fourth (ocf::pacemaker:Dummy): Started virt-157 > * fifth (ocf::pacemaker:Dummy): Started virt-156 > > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Shutting down virt-156. > [root@virt-158 ~]# qarsh virt-156 pcs cluster stop > Stopping Cluster (pacemaker)... > Stopping Cluster (corosync)... Node virt-156 is offline, resources "third" and "fifth" are marked as (LOCKED). > [root@virt-158 ~]# pcs status > [...output omitted...] > Node List: > * Online: [ virt-157 virt-158 ] > * OFFLINE: [ virt-156 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-157 > * Clone Set: locking-clone [locking]: > * Started: [ virt-157 virt-158 ] > * Stopped: [ virt-156 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Stopped virt-156 (LOCKED) > * fourth (ocf::pacemaker:Dummy): Started virt-158 > * fifth (ocf::pacemaker:Dummy): Stopped virt-156 (LOCKED) > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Start cluster on the node again so that it rejoins the cluster. Locked resources should get started, not necessarily on the same node (due to eg. load balancing). > [root@virt-158 ~]# qarsh virt-156 pcs cluster start > Starting Cluster... Resources "third" and "fifth" are recovered on node virt-156. > [root@virt-158 ~]# pcs status > [...output omitted...] > Node List: > * Online: [ virt-156 virt-157 virt-158 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-156 > * Clone Set: locking-clone [locking]: > * Started: [ virt-156 virt-157 virt-158 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-156 > * fourth (ocf::pacemaker:Dummy): Started virt-158 > * fifth (ocf::pacemaker:Dummy): Started virt-156 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled CASE 3 ------- Property "shutdown-lock" is specified as true. Shutting down a node and running "pcs resource refresh <resource> node=<node>" (both resource and node must be specified) while the node is down and the resource is locked, will result in the lock being removed, and the resource starting elsewhere. Verify whether "shutdown-lock" property is true. > [root@virt-158 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: true > shutdown-lock-limit: 0 > > [root@virt-158 ~]# pcs status > [...output omitted...] > Node List: > * Online: [ virt-156 virt-157 virt-158 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-156 > * Clone Set: locking-clone [locking]: > * Started: [ virt-156 virt-157 virt-158 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-156 > * fourth (ocf::pacemaker:Dummy): Started virt-158 > * fifth (ocf::pacemaker:Dummy): Started virt-156 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Shutting down virt-156. > [root@virt-158 ~]# qarsh virt-156 pcs cluster stop > Stopping Cluster (pacemaker)... > Stopping Cluster (corosync)... Node virt-156 is offline, resources "third" and "fifth" are marked as (LOCKED). > [root@virt-158 ~]# pcs status > [...output omitted...] > Node List: > * Online: [ virt-157 virt-158 ] > * OFFLINE: [ virt-156 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-157 > * Clone Set: locking-clone [locking]: > * Started: [ virt-157 virt-158 ] > * Stopped: [ virt-156 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Stopped virt-156 (LOCKED) > * fourth (ocf::pacemaker:Dummy): Started virt-158 > * fifth (ocf::pacemaker:Dummy): Stopped virt-156 (LOCKED) > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Refreshing resource "third" and checking the status. Node virt-156 is still offline, resource "third" restarted on virt-158. > [root@virt-158 ~]# pcs resource refresh third node=virt-156 > Cleaned up third on virt-156 > * 'third' is locked to node virt-156 due to shutdown > Waiting for 1 reply from the controller. OK > [root@virt-158 ~]# pcs status > [...output omitted...] > Node List: > * Online: [ virt-157 virt-158 ] > * OFFLINE: [ virt-156 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-157 > * Clone Set: locking-clone [locking]: > * Started: [ virt-157 virt-158 ] > * Stopped: [ virt-156 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-158 > * fourth (ocf::pacemaker:Dummy): Started virt-158 > * fifth (ocf::pacemaker:Dummy): Stopped virt-156 (LOCKED) > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Refreshing resource "fifth" and checking the status. Node virt-156 is still offline, resource "fifth" restarted on virt-157. > [root@virt-158 ~]# pcs resource refresh fifth node=virt-156 > Cleaned up fifth on virt-156 > * 'fifth' is locked to node virt-156 due to shutdown > Waiting for 1 reply from the controller. OK > [root@virt-158 ~]# pcs status > [...output omitted...] > Node List: > * Online: [ virt-157 virt-158 ] > * OFFLINE: [ virt-156 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-157 > * Clone Set: locking-clone [locking]: > * Started: [ virt-157 virt-158 ] > * Stopped: [ virt-156 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-158 > * fourth (ocf::pacemaker:Dummy): Started virt-158 > * fifth (ocf::pacemaker:Dummy): Started virt-157 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Start cluster on the node again so that it rejoins the cluster. > [root@virt-158 ~]# qarsh virt-156 pcs cluster start > Starting Cluster... > [root@virt-158 ~]# pcs status > [...output omitted...] > Node List: > * Online: [ virt-156 virt-157 virt-158 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-156 > * Clone Set: locking-clone [locking]: > * Started: [ virt-156 virt-157 virt-158 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-156 > * fourth (ocf::pacemaker:Dummy): Started virt-158 > * fifth (ocf::pacemaker:Dummy): Started virt-157 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled CASE 4 ------- Property "shutdown-lock" is specified as true and "shutdown-lock-limit" is specified as an amount of time (5 min) and the node shutdown is initiated. After the "shutdown-lock-limit" expires, locked resources will get unlocked and started elsewhere, while the node remains down. In 7.8 this was also dependent on "cluster-recheck-interval" setting, in 8.2 the "cluster-recheck-interval" is automatically adjusted to account for "shutdown-lock" expiration and failure timeouts. See bz#1747553 for more details. Verify whether "shutdown-lock" is set to true and check the setting of "shutdown-lock-limit" property. > [root@virt-158 16:27:37 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: true > shutdown-lock-limit: 0 Set "shutdown-lock-limit" to 5 minutes and verify. > [root@virt-158 16:27:47 ~]# pcs property set shutdown-lock-limit=5min > [root@virt-158 16:27:55 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: true > shutdown-lock-limit: 5min > [root@virt-158 16:28:09 ~]# pcs status > Cluster name: STSRHTS26675 > Cluster Summary: > * Stack: corosync > * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum > * Last updated: Thu Jan 30 16:28:16 2020 > * Last change: Thu Jan 30 16:27:55 2020 by root via cibadmin on virt-158 > * 3 nodes configured > * 15 resource instances configured > Node List: > * Online: [ virt-156 virt-157 virt-158 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-156 > * Clone Set: locking-clone [locking]: > * Started: [ virt-156 virt-157 virt-158 ] > * first (ocf::pacemaker:Dummy): Started virt-156 > * second (ocf::pacemaker:Dummy): Started virt-157 > * third (ocf::pacemaker:Dummy): Started virt-158 > * fourth (ocf::pacemaker:Dummy): Started virt-156 > * fifth (ocf::pacemaker:Dummy): Started virt-157 > * sixth (ocf::pacemaker:Dummy): Started virt-158 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Shutting down node virt-156. > [root@virt-158 16:28:17 ~]# qarsh virt-156 pcs cluster stop > Stopping Cluster (pacemaker)... > Stopping Cluster (corosync)... Resources "first" and "fourth" on virt-156 are marked as (LOCKED), node virt-156 remains down. > [root@virt-158 16:28:29 ~]# pcs status > [...output omitted...] > Node List: > * Online: [ virt-157 virt-158 ] > * OFFLINE: [ virt-156 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-157 > * Clone Set: locking-clone [locking]: > * Started: [ virt-157 virt-158 ] > * Stopped: [ virt-156 ] > * first (ocf::pacemaker:Dummy): Stopped virt-156 (LOCKED) > * second (ocf::pacemaker:Dummy): Started virt-158 > * third (ocf::pacemaker:Dummy): Started virt-158 > * fourth (ocf::pacemaker:Dummy): Stopped virt-156 (LOCKED) > * fifth (ocf::pacemaker:Dummy): Started virt-157 > * sixth (ocf::pacemaker:Dummy): Started virt-158 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Waiting for 5 minutes. The resources have started on other nodes and virt-156 is still offline. > [root@virt-158 16:34:44 ~]# pcs status > Cluster name: STSRHTS26675 > Cluster Summary: > * Stack: corosync > * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum > * Last updated: Thu Jan 30 16:34:48 2020 > * Last change: Thu Jan 30 16:27:55 2020 by root via cibadmin on virt-158 > * 3 nodes configured > * 15 resource instances configured > Node List: > * Online: [ virt-157 virt-158 ] > * OFFLINE: [ virt-156 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-157 > * Clone Set: locking-clone [locking]: > * Started: [ virt-157 virt-158 ] > * Stopped: [ virt-156 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-158 > * third (ocf::pacemaker:Dummy): Started virt-157 > * fourth (ocf::pacemaker:Dummy): Started virt-157 > * fifth (ocf::pacemaker:Dummy): Started virt-158 > * sixth (ocf::pacemaker:Dummy): Started virt-158 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Start cluster on the node again so that it rejoins the cluster. > [root@virt-158 16:34:48 ~]# qarsh virt-156 pcs cluster start > Starting Cluster... > [root@virt-158 16:37:00 ~]# pcs status > Cluster name: STSRHTS26675 > Cluster Summary: > * Stack: corosync > * Current DC: virt-158 (version 2.0.3-4.el8-4b1f869f0f) - partition with quorum > * Last updated: Thu Jan 30 16:37:10 2020 > * Last change: Thu Jan 30 16:27:55 2020 by root via cibadmin on virt-158 > * 3 nodes configured > * 15 resource instances configured > Node List: > * Online: [ virt-156 virt-157 virt-158 ] > Full List of Resources: > * fence-virt-156 (stonith:fence_xvm): Started virt-157 > * fence-virt-157 (stonith:fence_xvm): Started virt-158 > * fence-virt-158 (stonith:fence_xvm): Started virt-156 > * Clone Set: locking-clone [locking]: > * Started: [ virt-156 virt-157 virt-158 ] > * first (ocf::pacemaker:Dummy): Started virt-158 > * second (ocf::pacemaker:Dummy): Started virt-156 > * third (ocf::pacemaker:Dummy): Started virt-157 > * fourth (ocf::pacemaker:Dummy): Started virt-157 > * fifth (ocf::pacemaker:Dummy): Started virt-158 > * sixth (ocf::pacemaker:Dummy): Started virt-156 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled marking verified in pacemaker-2.0.3-4.el8 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:1609 |