Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1988568

Summary: [RFE] Add ability to use sbd on cluster nodes but not remote nodes
Product: Red Hat Enterprise Linux 9 Reporter: Ken Gaillot <kgaillot>
Component: pacemakerAssignee: Klaus Wenninger <kwenning>
Status: CLOSED CURRENTRELEASE QA Contact: cluster-qe <cluster-qe>
Severity: medium Docs Contact:
Priority: high    
Version: 9.0CC: cfeist, cluster-maint, cluster-qe, dpeess, kgaillot, kwalker, kwenning, lmiccini, ltamagno, michele, mnovacek, msmazova, phagara, sbradley, slevine
Target Milestone: betaKeywords: FutureFeature, Triaged
Target Release: 9.0 BetaFlags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: pacemaker-2.1.0-11.el9 Doc Type: Enhancement
Doc Text:
Feature: Users can now configure watchdog-based SBD using the "fence_watchdog" agent, which allows configurations where only certain nodes use SBD. A cluster may only have a single such device, and it must be named "watchdog". Reason: Previously, to use watchdog-based SBD, all nodes in the cluster had to use it. That prevented using SBD in a cluster where some nodes support it but other nodes (often remote nodes) required some other form of fencing. Result: Users can configure clusters where only some nodes use watchdog-based SBD for fencing, and other nodes use other fencing types.
 Story Points: ---
Clone Of: 1443666 Environment:
Last Closed: 2021-12-07 21:57:54 UTC Type: Enhancement
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1443666    
Bug Blocks: 1757982    

Description Ken Gaillot 2021-07-30 20:25:17 UTC 
+++ This bug was initially created as a clone of Bug #1443666 +++

Pacemaker currently requires that sbd usage be configured for all nodes or none. While this makes sense for cluster nodes to ensure that any node can initiate fencing, Pacemaker Remote nodes do not initiate fencing, so they could potentially be configured with other fence devices without needing sbd.

This could be particularly useful if remote nodes are added from a pool of virtual machines (possibly via a cloud) that don't have hardware watchdogs, but do have another means of fencing.

--- Additional comment from Klaus Wenninger on 2018-04-16 13:16:58 UTC ---

As I'm currently working on it let me give you an outline of how I test the feature:

My cluster consists of node2, node3, node4, remote_node1.

As we've finally decided to go the route of making the fencing-device visible I have the following fencing-resource (note that the name has to be watchdog):

  Resource: watchdog (class=stonith type=fence_watchdog)
    Attributes: pcmk_host_list="node2 node3"
    Operations: monitor interval=60s (watchdog-monitor-interval-60s)

  stonith-watchdog-timeout: 30

This config makes watchdog-fencing available for node2 & node3 while
node4 & remote_node1 don't support being fenced via watchdog-fencing.

As long as you don't configure any other means of fencing you can now easily test the feature by simple ungracefully bringing down one of the 4 nodes.

Doing that with node2 or node3 - while watching the cluster with crm_mon - should show the node as unclean for 30 seconds and afterwards it should go to offline.
Doing the same with node4 or remote_node1 would make them stay unclean for eternity.
Comment 2 Ken Gaillot 2021-08-04 22:25:45 UTC 
Fixed upstream by commits b49f495 5dd1e44 53dd360
Comment 6 Markéta Smazová 2021-08-30 20:12:30 UTC 
    [root@virt-542 ~]# rpm -q pacemaker
    pacemaker-2.1.0-11.el9.x86_64
    
    [root@virt-542 ~]# rpm -q sbd
    sbd-1.5.0-2.el9.1.x86_64

Setup 3-node cluster with 1 remote node:

    [root@virt-542 ~]# pcs status --full
    Cluster name: test_cluster
    Cluster Summary:
      * Stack: corosync
      * Current DC: virt-543 (2) (version 2.1.0-11.el9-7c3f660707) - partition with quorum
      * Last updated: Mon Aug 30 21:50:16 2021
      * Last change:  Mon Aug 30 21:48:24 2021 by root via cibadmin on virt-542
      * 4 nodes configured
      * 2 resource instances configured

    Node List:
      * Online: [ virt-542 (1) virt-543 (2) virt-558 (3) ]
      * RemoteOnline: [ virt-559 ]

    Full List of Resources:
      * virt-559	(ocf:pacemaker:remote):	 Started virt-542
      * watchdog	(stonith:fence_watchdog):	 Started virt-543

    Migration Summary:

    Tickets:

    PCSD Status:
      virt-542: Online
      virt-543: Online
      virt-558: Online

    Daemon Status:
      corosync: active/enabled
      pacemaker: active/enabled
      pcsd: active/enabled
      sbd: active/enabled

Fence watchdog is configured and sbd is running on only two of the cluster nodes ("virt-542 virt-543"):

    [root@virt-542 ~]# pcs stonith config watchdog
     Resource: watchdog (class=stonith type=fence_watchdog)
      Attributes: pcmk_host_list="virt-542 virt-543"
      Operations: monitor interval=60s (watchdog-monitor-interval-60s)

    [root@virt-542 ~]# pcs stonith sbd status
    SBD STATUS
    <node name>: <installed> | <enabled> | <running>
    virt-558: YES |  NO |  NO
    virt-542: YES | YES | YES
    virt-543: YES | YES | YES

killall -9 corosync on the node virt-543, which has fence_watchdog configured :

    [root@virt-543 ~]# killall -9 corosync

Node becomes unclean:

    [root@virt-542 ~]# crm_mon -1rfm
    Cluster Summary:
      * Stack: corosync
      * Current DC: virt-542 (version 2.1.0-11.el9-7c3f660707) - partition with quorum
      * Last updated: Mon Aug 30 21:52:43 2021
      * Last change:  Mon Aug 30 21:48:24 2021 by root via cibadmin on virt-542
      * 4 nodes configured
      * 2 resource instances configured

    Node List:
      * Node virt-543: UNCLEAN (offline)
      * Online: [ virt-542 virt-558 ]
      * RemoteOnline: [ virt-559 ]

    Full List of Resources:
      * virt-559	(ocf:pacemaker:remote):	 Started virt-542
      * watchdog	(stonith:fence_watchdog):	 Started [ virt-558 virt-543 ]

    Migration Summary:

    Fencing History:
      * reboot of virt-543 pending: client=pacemaker-controld.59607, origin=virt-542

Node is fenced:

    [root@virt-542 ~]# crm_mon -1rfm
    Cluster Summary:
      * Stack: corosync
      * Current DC: virt-542 (version 2.1.0-11.el9-7c3f660707) - partition with quorum
      * Last updated: Mon Aug 30 21:52:54 2021
      * Last change:  Mon Aug 30 21:48:24 2021 by root via cibadmin on virt-542
      * 4 nodes configured
      * 2 resource instances configured

    Node List:
      * Online: [ virt-542 virt-558 ]
      * OFFLINE: [ virt-543 ]
      * RemoteOnline: [ virt-559 ]

    Full List of Resources:
      * virt-559	(ocf:pacemaker:remote):	 Started virt-542
      * watchdog	(stonith:fence_watchdog):	 Started virt-558

    Migration Summary:

    Fencing History:
      * reboot of virt-543 successful: delegate=virt-542, client=pacemaker-controld.59607, origin=virt-542, last-successful='2021-08-30 21:52:52 +02:00'


Node is rebooted again:

    [root@virt-542 ~]# crm_mon -1rfm
    Cluster Summary:
      * Stack: corosync
      * Current DC: virt-542 (version 2.1.0-11.el9-7c3f660707) - partition with quorum
      * Last updated: Mon Aug 30 21:53:51 2021
      * Last change:  Mon Aug 30 21:48:24 2021 by root via cibadmin on virt-542
      * 4 nodes configured
      * 2 resource instances configured

    Node List:
      * Online: [ virt-542 virt-543 virt-558 ]
      * RemoteOnline: [ virt-559 ]

    Full List of Resources:
      * virt-559	(ocf:pacemaker:remote):	 Started virt-542
      * watchdog	(stonith:fence_watchdog):	 Started virt-558

    Migration Summary:

    Fencing History:
      * reboot of virt-543 successful: delegate=virt-542, client=pacemaker-controld.59607, origin=virt-542, last-successful='2021-08-30 21:52:52 +02:00'

killall -9 corosync on the node virt-558, which doesn't have fence_watchdog configured:

    [root@virt-558 ~]# killall -9 corosync


Node is marked as UNCLEAN, fencing failed:

    [root@virt-542 ~]# crm_mon -1rfm
    Cluster Summary:
      * Stack: corosync
      * Current DC: virt-542 (version 2.1.0-11.el9-7c3f660707) - partition with quorum
      * Last updated: Mon Aug 30 22:01:27 2021
      * Last change:  Mon Aug 30 21:48:24 2021 by root via cibadmin on virt-542
      * 4 nodes configured
      * 2 resource instances configured

    Node List:
      * Node virt-558: UNCLEAN (offline)
      * Online: [ virt-542 virt-543 ]
      * RemoteOnline: [ virt-559 ]

    Full List of Resources:
      * virt-559	(ocf:pacemaker:remote):	 Started virt-542
      * watchdog	(stonith:fence_watchdog):	 Started [ virt-558 virt-543 ]

    Migration Summary:

    Failed Fencing Actions:
      * reboot of virt-558 failed: delegate=, client=pacemaker-controld.59607, origin=virt-542, last-failed='2021-08-30 22:01:24 +02:00' 

    Fencing History:
      * reboot of virt-543 successful: delegate=virt-542, client=pacemaker-controld.59607, origin=virt-542, last-successful='2021-08-30 21:52:52 +02:00'


Node stays in the UNCLEAN state.


Verified as SanityOnly in pacemaker-2.1.0-11.el9