Bug 1781820
Summary: | Option to skip resource recovery after clean node shutdown (Zero Touch) | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Chris Feist <cfeist> |
Component: | pacemaker | Assignee: | Ken Gaillot <kgaillot> |
Status: | CLOSED ERRATA | QA Contact: | Markéta Smazová <msmazova> |
Severity: | medium | Docs Contact: | Steven J. Levine <slevine> |
Priority: | low | ||
Version: | 7.7 | CC: | cfeist, cluster-maint, cluster-qe, djansa, kgaillot, msmazova, sbradley, slevine, tborcin |
Target Milestone: | rc | ||
Target Release: | 7.8 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | pacemaker-1.1.21-4.el7 | Doc Type: | Enhancement |
Doc Text: |
.Pacemaker support for configuring resources to remain stopped on clean node shutdown
When a cluster node shuts down, Pacemaker’s default response is to stop all resources running on that node and recover them elsewhere. Some users prefer to have high availability only for failures, and to treat clean shutdowns as scheduled outages. To address this, Pacemaker now supports the `shutdown-lock` and `shutdown-lock-limit` cluster properties to specify that resources active on a node when it shuts down should remain stopped until the node next rejoins. Users can now use clean shutdowns as scheduled outages without any manual intervention. For information on configuring resources to remain stopped on a clean node shutdown, see link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/high_availability_add-on_reference/index?lb_target=production#s1-shutdown-lock-HAAR[Configuring Resources to Remain Stopped on Clean Node Shutdown].
|
Story Points: | --- |
Clone Of: | 1712584 | Environment: | |
Last Closed: | 2020-03-31 19:41:51 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1712584 | ||
Bug Blocks: |
Comment 4
Ken Gaillot
2020-01-16 20:35:53 UTC
QE: To test: 1. Configure a cluster of at least two nodes (so you can shut down one and retain quorum) with at least one resource. 2. If shutdown-lock is not specified or specified as false, shutting down a node running a resource will result in the resource being recovered on another node. 3. If shutdown-lock is specified as true, shutting down a node running a resource will result in the resource being stopped. - "pcs status" will show the resource as "LOCKED" while the node is down. - Starting the node again will result in the resource starting there again. (If the cluster has multiple resources, load balancing could result in it moving afterward, unless stickiness or a location preference is used.) - Running "pcs resource refresh <resource> --node <node>" while the node is down and the resource is locked will result in the lock being removed, and the resource starting elsewhere. - If shutdown-lock-limit is specified as an amount of time (e.g. 5min), then after that much time has passed since the shutdown was initiated (within the granularity of the cluster-recheck-interval), the resource will start elsewhere even if the node remains down. shutdown-lock will work for either cluster nodes or remote nodes, but not guest nodes. shutdown-lock-limit and manual refresh will work with remote nodes only if the remote node is stopped via "systemctl stop pacemaker_remote" on the remote node, followed by "pcs resource disable <remote-connection-resource>", followed by the lock expiration or refresh. CASE 1 ------- Property "shutdown-lock" is not specified or is specified as false. Shutting down a node running a resource will result in the resource being recovered on another node (current behavior). > [root@virt-003 ~]# rpm -q pacemaker > pacemaker-1.1.21-4.el7.x86_64 Verify whether "shutdown-lock" property is false. > [root@virt-003 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: false > shutdown-lock-limit: 0 There are two resources "second" and "fifth" on virt-004. > [root@virt-003 ~]# pcs status > Cluster name: STSRHTS2974 > Stack: corosync > Current DC: virt-003 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum > Last updated: Fri Jan 31 11:02:10 2020 > Last change: Fri Jan 31 10:56:40 2020 by root via cibadmin on virt-003 > 3 nodes configured > 15 resources configured > Online: [ virt-003 virt-004 virt-005 ] > Full list of resources: > > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-004 > fence-virt-005 (stonith:fence_xvm): Started virt-005 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-004 virt-005 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-004 virt-005 ] > first (ocf::pacemaker:Dummy): Started virt-003 > second (ocf::pacemaker:Dummy): Started virt-004 > third (ocf::pacemaker:Dummy): Started virt-005 > fourth (ocf::pacemaker:Dummy): Started virt-003 > fifth (ocf::pacemaker:Dummy): Started virt-004 > sixth (ocf::pacemaker:Dummy): Started virt-005 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Shutting down virt-004. > [root@virt-003 ~]# qarsh virt-004 pcs cluster stop > Stopping Cluster (pacemaker)... > Stopping Cluster (corosync)... Node virt-004 is offline, resources "second" and "fifth" are recovered on other nodes. > [root@virt-003 ~]# pcs status > Cluster name: STSRHTS2974 > Stack: corosync > Current DC: virt-003 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum > Last updated: Fri Jan 31 11:03:35 2020 > Last change: Fri Jan 31 10:56:40 2020 by root via cibadmin on virt-003 > 3 nodes configured > 15 resources configured > Online: [ virt-003 virt-005 ] > OFFLINE: [ virt-004 ] > Full list of resources: > > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-005 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > first (ocf::pacemaker:Dummy): Started virt-003 > second (ocf::pacemaker:Dummy): Started virt-003 > third (ocf::pacemaker:Dummy): Started virt-005 > fourth (ocf::pacemaker:Dummy): Started virt-003 > fifth (ocf::pacemaker:Dummy): Started virt-005 > sixth (ocf::pacemaker:Dummy): Started virt-005 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Start cluster on the node again so that it rejoins the cluster. > [root@virt-003 ~]# qarsh virt-004 pcs cluster start > Starting Cluster (corosync)... > Starting Cluster (pacemaker)... > [root@virt-003 ~]# pcs status > Cluster name: STSRHTS2974 > Stack: corosync > Current DC: virt-003 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum > Last updated: Fri Jan 31 11:04:00 2020 > Last change: Fri Jan 31 10:56:40 2020 by root via cibadmin on virt-003 > 3 nodes configured > 15 resources configured > Online: [ virt-003 virt-004 virt-005 ] > Full list of resources: > > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-004 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-004 virt-005 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-004 virt-005 ] > first (ocf::pacemaker:Dummy): Started virt-003 > second (ocf::pacemaker:Dummy): Started virt-004 > third (ocf::pacemaker:Dummy): Started virt-005 > fourth (ocf::pacemaker:Dummy): Started virt-003 > fifth (ocf::pacemaker:Dummy): Started virt-005 > sixth (ocf::pacemaker:Dummy): Started virt-004 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled CASE 2 ------- Property "shutdown-lock" is specified as true, shutting down a node running a resource will result in the resource being stopped. "pcs status" will show the resource as "LOCKED" while the node is down. Starting the node again will result in the resource starting there again. Set "shutdown-lock" to true and verify. > [root@virt-003 ~]# pcs property set shutdown-lock=true > [root@virt-003 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: true > shutdown-lock-limit: 0 There are resources "second" and "sixth" on virt-004. > [root@virt-003 ~]# pcs status > [...output omitted...] > Full list of resources: > > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-004 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-004 virt-005 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-004 virt-005 ] > first (ocf::pacemaker:Dummy): Started virt-003 > second (ocf::pacemaker:Dummy): Started virt-004 > third (ocf::pacemaker:Dummy): Started virt-005 > fourth (ocf::pacemaker:Dummy): Started virt-003 > fifth (ocf::pacemaker:Dummy): Started virt-005 > sixth (ocf::pacemaker:Dummy): Started virt-004 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Shutting down virt-004. > [root@virt-003 ~]# qarsh virt-004 pcs cluster stop > Stopping Cluster (pacemaker)... > Stopping Cluster (corosync)... Node virt-004 is offline, resources "second" and "sixth" are marked as (LOCKED). > [root@virt-003 ~]# pcs status > [...output omitted...] > Online: [ virt-003 virt-005 ] > OFFLINE: [ virt-004 ] > Full list of resources: > > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-003 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Stopped virt-004 (LOCKED) > third (ocf::pacemaker:Dummy): Started virt-005 > fourth (ocf::pacemaker:Dummy): Started virt-003 > fifth (ocf::pacemaker:Dummy): Started virt-005 > sixth (ocf::pacemaker:Dummy): Stopped virt-004 (LOCKED) > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Start cluster on the node again so that it rejoins the cluster. Locked resources should get started, not necessarily on the same node (due to eg. load balancing). > [root@virt-003 ~]# qarsh virt-004 pcs cluster start > Starting Cluster (corosync)... > Starting Cluster (pacemaker)... Resources "second" and "sixth" are recovered on virt-004. > [root@virt-003 ~]# pcs status > [...output omitted...] > Online: [ virt-003 virt-004 virt-005 ] > Full list of resources: > > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-004 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-004 virt-005 ] > Clone Set: clvmd-clone [clvmd] > clvmd (ocf::heartbeat:clvm): Starting virt-004 > Started: [ virt-003 virt-005 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-004 > third (ocf::pacemaker:Dummy): Started virt-003 > fourth (ocf::pacemaker:Dummy): Started virt-003 > fifth (ocf::pacemaker:Dummy): Started virt-005 > sixth (ocf::pacemaker:Dummy): Started virt-004 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled CASE 3 ------- Property "shutdown-lock" is specified as true. Shutting down a node and running "pcs resource refresh <resource> --node <node>" (both resource and node must be specified) while the node is down and the resource is locked, will result in the lock being removed, and the resource starting elsewhere. Verify whether "shutdown-lock" property is true. > [root@virt-003 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: true > shutdown-lock-limit: 0 > > [root@virt-003 ~]# pcs status > [...output omitted...] > Online: [ virt-003 virt-004 virt-005 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-004 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-004 virt-005 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-004 virt-005 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-004 > third (ocf::pacemaker:Dummy): Started virt-003 > fourth (ocf::pacemaker:Dummy): Started virt-003 > fifth (ocf::pacemaker:Dummy): Started virt-005 > sixth (ocf::pacemaker:Dummy): Started virt-004 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Shutting down virt-004. > [root@virt-003 ~]# qarsh virt-004 pcs cluster stop > Stopping Cluster (pacemaker)... > Stopping Cluster (corosync)... Node virt-004 is offline, resources "second" and "sixth" are marked as (LOCKED). > [root@virt-003 ~]# pcs status > [...output omitted...] > Online: [ virt-003 virt-005 ] > OFFLINE: [ virt-004 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-003 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Stopped virt-004 (LOCKED) > third (ocf::pacemaker:Dummy): Started virt-003 > fourth (ocf::pacemaker:Dummy): Started virt-005 > fifth (ocf::pacemaker:Dummy): Started virt-005 > sixth (ocf::pacemaker:Dummy): Stopped virt-004 (LOCKED) > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Refreshing resource "second", and checking the status. Node virt-004 is still offline, resource "second" restarted on virt-003. > [root@virt-003 ~]# pcs resource refresh second --node virt-004 > Cleaned up second on virt-004 > * 'second' is locked to node virt-004 due to shutdown > Waiting for 1 reply from the CRMd. OK > [root@virt-003 ~]# pcs status > [...output omitted...] > Online: [ virt-003 virt-005 ] > OFFLINE: [ virt-004 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-003 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-003 > third (ocf::pacemaker:Dummy): Started virt-005 > fourth (ocf::pacemaker:Dummy): Started virt-005 > fifth (ocf::pacemaker:Dummy): Started virt-003 > sixth (ocf::pacemaker:Dummy): Stopped virt-004 (LOCKED) > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Refreshing resource "sixth" and checking the status. Node virt-004 is still offline, resource "sixth" restarted on virt-003. > [root@virt-003 ~]# pcs resource refresh sixth --node virt-004 > Cleaned up sixth on virt-004 > * 'sixth' is locked to node virt-004 due to shutdown > Waiting for 1 reply from the CRMd. OK > [root@virt-003 ~]# pcs status > [...output omitted...] > Online: [ virt-003 virt-005 ] > OFFLINE: [ virt-004 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-003 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-003 > third (ocf::pacemaker:Dummy): Started virt-005 > fourth (ocf::pacemaker:Dummy): Started virt-005 > fifth (ocf::pacemaker:Dummy): Started virt-003 > sixth (ocf::pacemaker:Dummy): Started virt-003 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Start cluster on the node again so that it rejoins the cluster. > [root@virt-003 ~]# qarsh virt-004 pcs cluster start > Starting Cluster (corosync)... > Starting Cluster (pacemaker)... > [root@virt-003 ~]# pcs status > [...output omitted...] > Online: [ virt-003 virt-004 virt-005 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-004 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-004 virt-005 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-004 virt-005 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-003 > third (ocf::pacemaker:Dummy): Started virt-004 > fourth (ocf::pacemaker:Dummy): Started virt-005 > fifth (ocf::pacemaker:Dummy): Started virt-003 > sixth (ocf::pacemaker:Dummy): Started virt-004 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled CASE 4 ------- Property "shutdown-lock" is specified as true, "shutdown-lock-limit" is specified as an amount of time and the node shutdown is initiated. After the "shutdown-lock-limit" expires, locked resources will get unlocked and started elsewhere, while the node remains down. It can take up to one "cluster-recheck-interval" longer than the configured "shutdown-lock-limit" to start the resource recovery (worst-case scenario, RHEL-8.2's "dynamic cluster recheck interval" feature removes this limitation). Verify whether "shutdown-lock" property is true and check the setting of "shutdown-lock-limit" property. > [root@virt-003 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: true > shutdown-lock-limit: 0 Set "shutdown-lock-limit" to 5 minutes and verify. > [root@virt-003 ~]# pcs property set shutdown-lock-limit=5min > [root@virt-003 ~]# pcs property list --all | grep shutdown-lock > shutdown-lock: true > shutdown-lock-limit: 5min Verify "cluster-recheck-interval" setting > [root@virt-003 ~]# pcs property list --all | grep cluster-recheck-interval > cluster-recheck-interval: 15min > [root@virt-003 11:58:24 ~]# pcs status > Cluster name: STSRHTS2974 > Stack: corosync > Current DC: virt-003 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum > Last updated: Fri Jan 31 11:58:33 2020 > Last change: Fri Jan 31 11:58:14 2020 by root via cibadmin on virt-003 > 3 nodes configured > 15 resources configured > Online: [ virt-003 virt-004 virt-005 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-004 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-004 virt-005 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-004 virt-005 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-003 > third (ocf::pacemaker:Dummy): Started virt-004 > fourth (ocf::pacemaker:Dummy): Started virt-005 > fifth (ocf::pacemaker:Dummy): Started virt-003 > sixth (ocf::pacemaker:Dummy): Started virt-004 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Shutting down node virt-004. > [root@virt-003 11:58:33 ~]# qarsh virt-004 pcs cluster stop > Stopping Cluster (pacemaker)... > Stopping Cluster (corosync)... Verifying that the node is down, and resources are marked as (LOCKED). > [root@virt-003 11:58:54 ~]# pcs status > Cluster name: STSRHTS2974 > Stack: corosync > Current DC: virt-003 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum > Last updated: Fri Jan 31 11:58:59 2020 > Last change: Fri Jan 31 11:58:14 2020 by root via cibadmin on virt-003 > 3 nodes configured > 15 resources configured > Online: [ virt-003 virt-005 ] > OFFLINE: [ virt-004 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-003 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-003 > third (ocf::pacemaker:Dummy): Stopped virt-004 (LOCKED) > fourth (ocf::pacemaker:Dummy): Started virt-005 > fifth (ocf::pacemaker:Dummy): Started virt-003 > sixth (ocf::pacemaker:Dummy): Stopped virt-004 (LOCKED) > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled After at most "shutdown-lock-limit" + "cluster-recheck-interval", the resources are recovered on other nodes and the original node remains down. > [root@virt-003 12:03:59 ~]# pcs status > Cluster name: STSRHTS2974 > Stack: corosync > Current DC: virt-003 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum > Last updated: Fri Jan 31 12:04:02 2020 > Last change: Fri Jan 31 11:58:14 2020 by root via cibadmin on virt-003 > 3 nodes configured > 15 resources configured > Online: [ virt-003 virt-005 ] > OFFLINE: [ virt-004 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-003 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-005 ] > Stopped: [ virt-004 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-003 > third (ocf::pacemaker:Dummy): Started virt-005 > fourth (ocf::pacemaker:Dummy): Started virt-005 > fifth (ocf::pacemaker:Dummy): Started virt-003 > sixth (ocf::pacemaker:Dummy): Started virt-003 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled Start cluster on the node again so that it rejoins the cluster. > [root@virt-003 12:07:03 ~]# qarsh virt-004 pcs cluster start > Starting Cluster (corosync)... > Starting Cluster (pacemaker)... > [root@virt-003 12:07:13 ~]# pcs status > Cluster name: STSRHTS2974 > Stack: corosync > Current DC: virt-003 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum > Last updated: Fri Jan 31 12:07:25 2020 > Last change: Fri Jan 31 11:58:14 2020 by root via cibadmin on virt-003 > 3 nodes configured > 15 resources configured > Online: [ virt-003 virt-004 virt-005 ] > Full list of resources: > fence-virt-003 (stonith:fence_xvm): Started virt-003 > fence-virt-004 (stonith:fence_xvm): Started virt-005 > fence-virt-005 (stonith:fence_xvm): Started virt-004 > Clone Set: dlm-clone [dlm] > Started: [ virt-003 virt-004 virt-005 ] > Clone Set: clvmd-clone [clvmd] > Started: [ virt-003 virt-004 virt-005 ] > first (ocf::pacemaker:Dummy): Started virt-005 > second (ocf::pacemaker:Dummy): Started virt-003 > third (ocf::pacemaker:Dummy): Started virt-004 > fourth (ocf::pacemaker:Dummy): Started virt-005 > fifth (ocf::pacemaker:Dummy): Started virt-003 > sixth (ocf::pacemaker:Dummy): Started virt-004 > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled marking verified in pacemaker-1.1.21-4.el7 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1032 |