Bug 2055935

Description of problem:

This is a big-time edge case, and we don't even support clusters with stonith-enabled=false, so this would be more for upstream. This is interesting behavior though. I'm raising it as an RHBZ rather than CLBZ only because a customer encountered it while troubleshooting a stonith device.

If a stonith device is created while stonith-enabled=false, and that stonith device fails to start, then `pcs stonith fence`/stonith_admin doesn't find it when it looks for devices capable of fencing a node.

In contrast, stonith_admin **does** find it in the following scenarios:
  - the device starts successfully while stonith-enabled=false; or
  - the device is created while stonith-enabled=true, it immediately fails to start, and then stonith-enabled is set to false before calling stonith_admin.


Brief demo:

[root@fastvm-rhel-8-0-23 ~]# pcs stonith config
[root@fastvm-rhel-8-0-23 ~]# pcs property set stonith-enabled=false
[root@fastvm-rhel-8-0-23 ~]# pcs stonith create vmfence fence_vmware_soap ip=1.1.1.1 login=asdf passwd=asdf pcmk_host_map='node1:node1;node2:node2'

[root@fastvm-rhel-8-0-23 ~]# pcs stonith status
  * vmfence	(stonith:fence_vmware_soap):	 Stopped

[root@fastvm-rhel-8-0-23 ~]# crm_mon --one-shot -U all -I failures
Failed Resource Actions:
  * vmfence_start_0 on node1 'error' (1): call=57, status='complete', exitreason='', last-rc-change='2022-02-17 16:29:12 -08:00', queued=0ms, exec=1649ms

[root@fastvm-rhel-8-0-23 ~]# pcs stonith fence node2 & tail -f /var/log/messages
...
Feb 17 16:32:48 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: Client stonith_admin.409161 wants to fence (reboot) node2 using any device
Feb 17 16:32:48 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: Requesting peer fencing (reboot) targeting node2
Feb 17 16:32:48 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: Couldn't find anyone to fence (reboot) node2 using any device
Feb 17 16:32:48 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: error: Operation 'reboot' targeting node2 by unknown node for stonith_admin.409161@node1: No such device
Feb 17 16:32:48 fastvm-rhel-8-0-23 pacemaker-controld[1406]: notice: Peer node2 was not terminated (reboot) by <anyone> on behalf of stonith_admin.409161: No such device


For brevity, I'm not including the demos of the two scenarios mentioned above where the device is found. I can demonstrate if needed.


There is a related issue that is arguably a bit worse: an old (possibly working) stonith device may appear in the list of available devices in this kind of scenario. In the demo below, the old working stonith device is xvm. I set stonith-enabled=false, deleted xvm, created vmfence, and ran pcs stonith fence. The fencer found xvm and used it to reboot node2.

[root@fastvm-rhel-8-0-23 ~]# pcs stonith config
[root@fastvm-rhel-8-0-23 ~]# pcs stonith create xvm fence_xvm pcmk_host_map='node1:fastvm-rhel-8.0-23;node2:fastvm-rhel-8.0-24'
[root@fastvm-rhel-8-0-23 ~]# pcs stonith status
  * xvm	(stonith:fence_xvm):	 Started node1

[root@fastvm-rhel-8-0-23 ~]# pcs property set stonith-enabled=false
[root@fastvm-rhel-8-0-23 ~]# pcs stonith delete xvm
Attempting to stop: xvm... Stopped

[root@fastvm-rhel-8-0-23 ~]# pcs stonith create vmfence fence_vmware_soap ip=1.1.1.1 login=asdf passwd=asdf pcmk_host_map='node1:node1;node2:node2'
[root@fastvm-rhel-8-0-23 ~]# pcs stonith status
  * vmfence	(stonith:fence_vmware_soap):	 Stopped
[root@fastvm-rhel-8-0-23 ~]# crm_mon --one-shot -U all -I failures
Failed Resource Actions:
  * vmfence_start_0 on node1 'error' (1): call=73, status='complete', exitreason='', last-rc-change='2022-02-17 16:36:34 -08:00', queued=0ms, exec=1599ms

[root@fastvm-rhel-8-0-23 ~]# pcs stonith fence node2 & tail -f /var/log/messages 
[1] 409343
...
Feb 17 16:36:45 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: Client stonith_admin.409345 wants to fence (reboot) node2 using any device
Feb 17 16:36:45 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: Requesting peer fencing (reboot) targeting node2
Feb 17 16:36:45 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: xvm is eligible to fence (reboot) node2 (aka. 'fastvm-rhel-8.0-24'): static-list
Feb 17 16:36:45 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: Requesting that node1 perform 'reboot' action targeting node2
Feb 17 16:36:45 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: xvm is eligible to fence (reboot) node2 (aka. 'fastvm-rhel-8.0-24'): static-list
Node: node2 fenced
Feb 17 16:36:47 fastvm-rhel-8-0-23 fence_xvm[409346]: Domain "fastvm-rhel-8.0-24" is ON
Feb 17 16:36:47 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: Operation 'reboot' [409346] (call 2 from stonith_admin.409345) targeting node2 using xvm returned 0 (OK)
Feb 17 16:36:47 fastvm-rhel-8-0-23 pacemaker-fenced[1402]: notice: Operation 'reboot' targeting node2 by node1 for stonith_admin.409345@node1: OK
Feb 17 16:36:47 fastvm-rhel-8-0-23 pacemaker-controld[1406]: notice: Peer node2 was terminated (reboot) by node1 on behalf of stonith_admin.409345: OK

-----

Version-Release number of selected component (if applicable):

pacemaker-2.1.0-8.el8

-----

How reproducible:

Always

-----

Steps to Reproduce:

1. Start with no stonith devices configured and stonith-enabled=true.
2. `pcs property set stonith-enabled=false`
3. Create a stonith device that's configured so that it will fail to start, and with pcmk_host_map or pcmk_host_list configured so that it's capable of fencing node2.
4. `pcs stonith fence node2`

-----

Actual results:

The stonith device is not found by the fencer as capable of fencing node2.

-----

Expected results:

The stonith device is found by the fencer as capable of fencing node2.

-----

Additional info:

The customer who encountered this is on RHEL 7, but there's no reason to consider fixing it there.