Bug 2058246

+++ This bug was initially created as a clone of Bug #1954099 +++

Description of problem:
Having sbd poison pill fencing configured together with stonith-watchdog-timeout property can be dangerous. When the property is set to a positive value, pacemaker would consider a potential fencing action successful even in situations when the action actually cannot be taken.

Version-Release number of selected component (if applicable):
pcs-0.10.8-1.el8.x86_64
sbd-1.4.2-2.el8.x86_64

How reproducible:
always

Steps to Reproduce:

1. setup a 3-node cluster, setup sbd with 1 disk and 1 fence_sbd agent 

[root@virt-242 ~]# pcs stonith sbd config
SBD_DELAY_START=no
SBD_STARTMODE=always
SBD_WATCHDOG_TIMEOUT=5

Watchdogs:
  virt-243: /dev/watchdog
  virt-242: /dev/watchdog
  virt-244: /dev/watchdog

Devices:
  virt-243: "/dev/disk/by-id/scsi-36001405d4c7b00480b44c6a824e5aa61"
  virt-242: "/dev/disk/by-id/scsi-36001405d4c7b00480b44c6a824e5aa61"
  virt-244: "/dev/disk/by-id/scsi-36001405d4c7b00480b44c6a824e5aa61"

[root@virt-242 ~]# pcs resource create fence_sbd_agent devices=/dev/disk/by-id/scsi-36001405d4c7b00480b44c6a824e5aa61

[root@virt-242 ~]# pcs status
Cluster name: STSRHTS9643
Cluster Summary:
  * Stack: corosync
  * Current DC: virt-243 (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Tue Apr 27 15:34:14 2021
  * Last change:  Tue Apr 27 15:34:08 2021 by root via cibadmin on virt-242
  * 3 nodes configured
  * 4 resource instances configured

Node List:
  * Online: [ virt-242 virt-243 virt-244 ]

Full List of Resources:
  * d1	(ocf::pacemaker:Dummy):	 Started virt-242
  * d2	(ocf::pacemaker:Dummy):	 Started virt-244
  * d3	(ocf::pacemaker:Dummy):	 Started virt-243
  * fence_sbd_agent	(stonith:fence_sbd):	 Started virt-242

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
  sbd: active/enabled

2. set stonith-watchdog-timeout>0 & restart cluster

[root@virt-242 ~]# pcs property set stonith-watchdog-timeout=10
[root@virt-242 ~]# pcs property
Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: STSRHTS9643
 dc-version: 2.0.5-9.el8-ba59be7122
 have-watchdog: true
 no-quorum-policy: freeze
 stonith-watchdog-timeout: 10

3. fail fence_sbd agent & block corosync on one node

[root@virt-242 ~]# pcs resource update fence_sbd_agent devices=/dev/disk/by-id/InvadidID
[root@virt-242 ~]# ip6tables -A INPUT ! -i lo -p udp --dport 5404 -j DROP && ip6tables -A INPUT ! -i lo -p udp --dport 5405 -j DROP && ip6tables -A OUTPUT ! -o lo -p udp --sport 5404 -j DROP && ip6tables -A OUTPUT ! -o lo -p udp --sport 5405 -j DROP

[root@virt-243 ~]# tail -f /var/log/messages
Apr 27 15:46:46 virt-243 pacemaker-fenced[654268]: notice: Couldn't find anyone to fence (reboot) virt-242 with any device
Apr 27 15:46:46 virt-243 pacemaker-fenced[654268]: notice: Waiting 10s for virt-242 to self-fence (reboot) for client pacemaker-controld.654272.c37a5c14
Apr 27 15:46:56 virt-243 pacemaker-fenced[654268]: notice: Self-fencing (reboot) by virt-242 for pacemaker-controld.654272.c37a5c14-4f5a-462c-bbe2-f63df1dfa562 assumed complete
Apr 27 15:46:56 virt-243 pacemaker-fenced[654268]: notice: Operation 'reboot' targeting virt-242 on virt-243 for pacemaker-controld.654272: OK
Apr 27 15:46:56 virt-243 pacemaker-controld[654272]: notice: Stonith operation 2/1:0:0:d7152a57-779d-4cd6-8158-7ab4cf5c4457: OK (0)
Apr 27 15:46:56 virt-243 pacemaker-controld[654272]: notice: Peer virt-242 was terminated (reboot) by virt-243 on behalf of pacemaker-controld.654272: OK

[root@virt-243 ~]# pcs stonith history
reboot of virt-242 successful: delegate=virt-243, client=pacemaker-controld.654272, origin=virt-243, completed='2021-04-27 15:46:56 +02:00'
1 event found

> No actual reboot

[root@virt-243 ~]# pcs status
...
Node List:
  * Online: [ virt-243 virt-244 ]
  * OFFLINE: [ virt-242 ]

Full List of Resources:
  * d1	(ocf::pacemaker:Dummy):	 Started virt-243
  * d2	(ocf::pacemaker:Dummy):	 Started virt-244
  * d3	(ocf::pacemaker:Dummy):	 Started virt-244
  * fence_sbd_agent	(stonith:fence_sbd):	 Stopped
..


Actual results:
The blocked node is considered rebooted, but it's actually not as fence_sbd is stopped and the disk is still available. Resource from the node is rescheduled.

Expected results:
For sbd with shared disk(s) pcs could forbid to have stonith-watchdog-timeout property set to positive values if there is no other case where this would be a desired configuration

Additional info:
Originally discovered in bz1952140

--- Additional comment from Tomas Jelinek on 2021-04-29 10:23:59 CEST ---

There are cluster properties managed automatically by pacemaker or pcs: cluster-infrastructure, cluster-name, dc-version, have-watchdog, last-lrm-refresh, stonith-watchdog-timeout. The 'pcs property' commands could check if users are trying to modify those, print an error message saying those are managed automatically, and require --force to proceed changing them.

--- Additional comment from Nina Hostakova on 2021-04-29 14:30:16 CEST ---

This approach would probably not work for watchdog-only sbd fencing where, on the contrary, the stonith-watchdog-timeout property needs to be set up manually so that watchdog fencing works properly.

--- Additional comment from Nina Hostakova on 2021-04-30 14:18:16 CEST ---

Digging more into stonith-watchdog-timeout, we have also found issues when configuring the property with pcs for sbd watchdog fencing (no disks):


1. When setting up stonith-watchdog-timeout property, the value needs to exceed SBD_WATCHDOG_TIMEOUT. If it doesn't, the cluster will end up in a fencing loop on all nodes (if cluster enabled). 'pcs property set' should give an error and forbid to set it instead

[root@virt-247 ~]# pcs stonith sbd config
SBD_DELAY_START=no
SBD_STARTMODE=always
SBD_WATCHDOG_TIMEOUT=5

Watchdogs:
  virt-247: /dev/watchdog
  virt-246: /dev/watchdog
  virt-248: /dev/watchdog

[root@virt-247 ~]# pcs property show --all | grep stonith-watchdog-timeout
 stonith-watchdog-timeout: 0

[root@virt-247 ~]# pcs property set stonith-watchdog-timeout=3

Broadcast message from systemd-journald.lab.eng.brq.redhat.com (Fri 2021-04-30 11:59:31 CEST):

pacemaker-controld[7343]:  emerg: Shutting down: stonith-watchdog-timeout (3) too short (must be >5000ms)


2. If stonith-watchdog-timeout property is set to a negative number, pcs rejects this value with an error (--force need to be used), even though it should be a supported configuration:

# man pacemaker-controld
...
If `stonith-watchdog-timeout` is set to a negative value, and
`SBD_WATCHDOG_TIMEOUT` is set, twice that value will be used.
...
[root@virt-023 ~]# pcs property set stonith-watchdog-timeout=-1
Error: invalid value of property: 'stonith-watchdog-timeout=-1', (use --force to override)
[root@virt-023 ~]# echo $?
1


The scope of the original bz is only to prevent misconfig for sbd with disks. So now the question is, if to use this bz for fixing overal pcs validation of the stonith-watchdog-timeout property or to create separate bzs for individual issues. Depends which approach works better for pcs.

--- Additional comment from Klaus Wenninger on 2021-04-30 15:56:22 CEST ---

Please leave it as is - at least for now - that pcs rejects setting stonith-watchdog-timeout=-1.
I consider this feature coming with pacemaker 2 highly dangerous - at least as implemented right now.
When watchdog-timeouts in SBD are configured differently on different nodes you easily get
into dangerous situations. Point is that the node that if you want to derive stonith-watchdog-timeout
used from the watchdog-timeout configured with SBD you would have to do that with the value
on the node that is to be fenced and not with the value read on the node that wants to fence.
Of course that all doesn't make a difference as long as watchdog-timeout is the same on all nodes
but atm there is nothing that would check verify that. All we have atm is that test on all nodes -
before resources are started and with every change of stonith-watchdog-timeout - if 
stonith-watchdog-timeout is compatible with local SBD settings. This approach works just if 
stonith-watchdog-timeout is a fixed value.

--- Additional comment from Tomas Jelinek on 2021-05-06 11:15:20 CEST ---

To summarize things up:
* When SBD is used without devices, then stonith-watchdog-timeout must be set to a value greater than SBD_WATCHDOG_TIMEOUT. This cannot be done automatically by pcs, as the property must be set after the cluster is restarted. The restart is not done automatically by pcs and is left to be done by users so that it does not disrupt cluster operation.
* When SBD is used with devices, then stonith-watchdog-timeout must not be set to value greater than 0.

Action items:
When stonith-watchdog-timeout property is being set by a user, check whether SBD is used with or without devices. If devices are used, prevent the property to be set if its value is not 0 or empty. If devices are not used, prevent the property to be set if its value is not greater than SBD_WATCHDOG_TIMEOUT. If SBD is not used at all, prevent the property to be set if its value is not 0 or empty.

--- Additional comment from Nina Hostakova on 2021-05-06 13:39:58 CEST ---

Tomas, thanks for summing up, that is exactly what we think should be done.