+++ This bug was initially created as a clone of Bug #2177996 +++ Description of problem: ###Similar to Bug 2024522, only difference is that we are using fence_scsi agent to build our cluster: "cmd": "pcs stonith create emc_fence fence_scsi \"devices=/dev/mapper/368ccf09800de023572e4a63b259687d1,/dev/mapper/368ccf098003e693c025c28dd5cdf69d9,/dev/mapper/368ccf0980076ee9888c1334880ea05d1\" \"pcmk_host_list=e2e-l4-236128,e2e-l4-236129\" pcmk_monitor_action=\"metadata\" pcmk_reboot_action=\"off\" pcmk_host_check=\"static-list\" meta provides=\"unfencing --force\"\n", [root@e2e-l4-236129 ~]# pcs stonith config Resource: emc_fence (class=stonith type=fence_scsi) Attributes: emc_fence-instance_attributes devices=/dev/mapper/368ccf098003e693c025c28dd5cdf69d9,/dev/mapper/368ccf0980076ee9888c1334880ea05d1,/dev/mapper/368ccf09800de023572e4a63b259687d1 pcmk_host_check=static-list pcmk_host_list=e2e-l4-236128,e2e-l4-236129 pcmk_monitor_action=metadata pcmk_reboot_action=off Meta Attributes: emc_fence-meta_attributes provides=unfencing Operations: monitor: emc_fence-monitor-interval-60s interval=60s ####Create several resources on the cluster: [root@e2e-l4-236129 ~]# pcs status Cluster name: RHCS Status of pacemakerd: 'Pacemaker is running' (last updated 2023-03-14 03:51:09 -04:00) Cluster Summary: * Stack: corosync * Current DC: e2e-l4-236128 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum * Last updated: Tue Mar 14 03:51:10 2023 * Last change: Tue Mar 14 03:27:11 2023 by root via cibadmin on e2e-l4-236128 * 2 nodes configured * 13 resource instances configured Node List: * Online: [ e2e-l4-236128 e2e-l4-236129 ] Full List of Resources: * emc_fence (stonith:fence_scsi): Started e2e-l4-236128 * Clone Set: dlm-clone [dlm]: * Started: [ e2e-l4-236128 e2e-l4-236129 ] * Clone Set: lvmlockd-clone [lvmlockd]: * Started: [ e2e-l4-236128 e2e-l4-236129 ] * Clone Set: alua_1_vg_1678695957-clone [alua_1_vg_1678695957]: * Started: [ e2e-l4-236128 e2e-l4-236129 ] * Clone Set: alua_0_vg_1678695957-clone [alua_0_vg_1678695957]: * Started: [ e2e-l4-236128 e2e-l4-236129 ] Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled ####And then we use update-scsi-devices to add or remove one more device to stonith devices [root@e2e-l4-236128 ~]# pcs stonith update-scsi-devices emc_fence remove /dev/mapper/368ccf09800b6f39e1227c2c42c3fb481 [root@e2e-l4-236128 ~]# pcs stonith update-scsi-devices emc_fence add /dev/mapper/368ccf09800b6f39e1227c2c42c3fb481 ####We can observe resource lock down and cluster restart Mar 14 03:25:29 e2e-l4-236128 pacemaker-schedulerd[2594]: notice: Unfencing Pacemaker Remote node e2e-l4-236129 because the definition of emc_fence changed Mar 14 03:25:29 e2e-l4-236128 pacemaker-schedulerd[2594]: notice: Unfencing Pacemaker Remote node e2e-l4-236128 because the definition of emc_fence changed Mar 14 03:25:29 e2e-l4-236128 pacemaker-schedulerd[2594]: notice: * Fence (on) e2e-l4-236128 Mar 14 03:25:29 e2e-l4-236128 pacemaker-schedulerd[2594]: notice: * Fence (on) e2e-l4-236129 Mar 14 03:25:29 e2e-l4-236128 pacemaker-schedulerd[2594]: notice: Actions: Restart emc_fence ( e2e-l4-236128 ) due to required stonith Mar 14 03:25:29 e2e-l4-236128 pacemaker-schedulerd[2594]: notice: Actions: Restart dlm:0 ( e2e-l4-236128 ) due to required stonith ####Is there anyway we can avoid restarting fencing when we modify the stonith devices? --- Additional comment from Miroslav Lisik on 2023-03-14 14:18:12 CET --- Ken, could you look at this? It does not work with latest pacemaker-2.1.5-7.el9 in RHEL-9.3. The last working version is pacemaker-2.1.4-2.el9 in RHEL-9.1. Is it possible that something has changed in pacemaker-2.1.4-3.el9 regarding digests calculation? (bz1872376) --- Additional comment from Ken Gaillot on 2023-03-14 17:21:18 CET --- (In reply to Miroslav Lisik from comment #1) > Ken, could you look at this? > > It does not work with latest pacemaker-2.1.5-7.el9 in RHEL-9.3. The last working version is pacemaker-2.1.4-2.el9 in RHEL-9.1. Is it possible that something has changed in pacemaker-2.1.4-3.el9 regarding digests calculation? (bz1872376) Not that I know of, and our regression test for it passes. Can you give me the CIB and commands you're using to test? --- Additional comment from Miroslav Lisik on 2023-03-14 18:08:44 CET --- Here are commands: export disk1=/dev/disk/by-id/scsi-SLIO-ORG_r91-disk-01_7ad95d75-3cf3-448e-a591-42b9ba690b22 export disk2=/dev/disk/by-id/scsi-SLIO-ORG_r91-disk-02_e9a0c17d-c631-41cf-a135-e3453ce0c501 pcs host auth -u hacluster -p password r91-1 r91-2 pcs cluster setup HACluster r91-1 r91-2 --start --wait pcs stonith create fence-scsi fence_scsi devices=$disk1 pcmk_host_check=static-list 'pcmk_host_list=r91-1 r91-2' pcmk_reboot_action=off meta provides=unfencing pcs resource create d1 ocf:pacemaker:Dummy pcs resource create d2 ocf:pacemaker:Dummy pcs cluster cib > cib_before.xml pcs stonith update-scsi-devices fence-scsi add $disk2 pcs cluster cib > cib_after.xml Logs after executing 'update-scsi-devices' command folllows. With latest rhel-9.1.z versions, see cib_before.xml, cib_after.xml: [root@r91-1 ~]# rpm -q pcs pacemaker pcs-0.11.3-4.el9_1.2.x86_64 pacemaker-2.1.4-5.el9_1.2.x86_64 [root@r91-1 ~]# journalctl -n0 -f Mar 14 17:36:51 r91-1 pacemaker-fenced[1663]: notice: Added 'fence-scsi' to device list (1 active device) Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Requesting local execution of stop operation for fence-scsi on r91-1 Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Result of stop operation for fence-scsi on r91-1: ok Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Requesting local execution of stop operation for d2 on r91-1 Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Result of stop operation for d2 on r91-1: ok Mar 14 17:36:51 r91-1 pacemaker-fenced[1663]: notice: fence-scsi is eligible to fence (on) r91-1: static-list Mar 14 17:36:51 r91-1 pacemaker-fenced[1663]: notice: fence-scsi is eligible to fence (on) r91-1: static-list Mar 14 17:36:51 r91-1 pacemaker-fenced[1663]: notice: Operation 'on' [2379] targeting r91-1 using fence-scsi returned 0 Mar 14 17:36:51 r91-1 pacemaker-fenced[1663]: notice: Operation 'on' targeting r91-1 by r91-1 for pacemaker-controld.1631@r91-2: OK (complete) Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: r91-1 was unfenced by r91-1 at the request of pacemaker-controld.1631@r91-2 Mar 14 17:36:51 r91-1 pacemaker-attrd[1665]: notice: Setting #node-unfenced[r91-1]: 1678811397 -> 1678811811 Mar 14 17:36:51 r91-1 pacemaker-attrd[1665]: notice: Setting #digests-all[r91-1]: fence-scsi:fence_scsi:ec9ecba84b274fb35effbf2a47226087, -> fence-scsi:fence_scsi:41f4daa097914fe0b3f6ba8363f28cf9, Mar 14 17:36:51 r91-1 pacemaker-attrd[1665]: notice: Setting #digests-secure[r91-1]: fence-scsi:fence_scsi:8a7469d06699bb5cdf0da9304affaf6e, -> fence-scsi:fence_scsi:2467afb1330d3c0048abdc371dea6bc3, Mar 14 17:36:51 r91-1 pacemaker-fenced[1663]: notice: Operation 'on' targeting r91-2 by r91-2 for pacemaker-controld.1631@r91-2: OK (complete) Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: r91-2 was unfenced by r91-2 at the request of pacemaker-controld.1631@r91-2 Mar 14 17:36:51 r91-1 pacemaker-attrd[1665]: notice: Setting #node-unfenced[r91-2]: 1678811397 -> 1678811811 Mar 14 17:36:51 r91-1 pacemaker-attrd[1665]: notice: Setting #digests-all[r91-2]: fence-scsi:fence_scsi:ec9ecba84b274fb35effbf2a47226087, -> fence-scsi:fence_scsi:41f4daa097914fe0b3f6ba8363f28cf9, Mar 14 17:36:51 r91-1 pacemaker-attrd[1665]: notice: Setting #digests-secure[r91-2]: fence-scsi:fence_scsi:8a7469d06699bb5cdf0da9304affaf6e, -> fence-scsi:fence_scsi:2467afb1330d3c0048abdc371dea6bc3, Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Requesting local execution of start operation for d2 on r91-1 Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Result of start operation for d2 on r91-1: ok Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Requesting local execution of start operation for fence-scsi on r91-1 Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Result of start operation for fence-scsi on r91-1: ok Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Requesting local execution of monitor operation for fence-scsi on r91-1 Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Requesting local execution of monitor operation for d2 on r91-1 Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Result of monitor operation for d2 on r91-1: ok Mar 14 17:36:51 r91-1 pacemaker-controld[1667]: notice: Result of monitor operation for fence-scsi on r91-1: ok [root@r91-2 ~]# journalctl -n0 -f Mar 14 17:36:38 r91-2 systemd[1]: systemd-hostnamed.service: Deactivated successfully. Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: State transition S_IDLE -> S_POLICY_ENGINE Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Added 'fence-scsi' to device list (1 active device) Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: Unfencing r91-2 (remote): because the definition of fence-scsi changed Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: Unfencing r91-1 (remote): because the definition of fence-scsi changed Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: * Fence (on) r91-1 Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: * Fence (on) r91-2 Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: Actions: Restart fence-scsi ( r91-1 ) due to required stonith Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: Actions: Restart d1 ( r91-2 ) due to required stonith Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: Actions: Restart d2 ( r91-1 ) due to required stonith Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: Calculated transition 5, saving inputs in /var/lib/pacemaker/pengine/pe-input-205.bz2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating stop operation fence-scsi_stop_0 on r91-1 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating stop operation d1_stop_0 locally on r91-2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Requesting local execution of stop operation for d1 on r91-2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating stop operation d2_stop_0 on r91-1 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Result of stop operation for d1 on r91-2: ok Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Requesting fencing (on) of node r91-2 Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Client pacemaker-controld.1631 wants to fence (on) r91-2 using any device Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Requesting peer fencing (on) targeting r91-2 Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: fence-scsi is eligible to fence (on) r91-2: static-list Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Requesting that r91-2 perform 'on' action targeting r91-2 Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: fence-scsi is eligible to fence (on) r91-2: static-list Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Requesting fencing (on) of node r91-1 Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Client pacemaker-controld.1631 wants to fence (on) r91-1 using any device Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Requesting peer fencing (on) targeting r91-1 Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Requesting that r91-1 perform 'on' action targeting r91-1 Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Operation 'on' targeting r91-1 by r91-1 for pacemaker-controld.1631@r91-2: OK (complete) Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Fence operation 10 for r91-1 passed Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: r91-1 was unfenced by r91-1 at the request of pacemaker-controld.1631@r91-2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating start operation fence-scsi_start_0 on r91-1 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating start operation d2_start_0 on r91-1 Mar 14 17:36:51 r91-2 pacemaker-attrd[1629]: notice: Setting #node-unfenced[r91-1]: 1678811397 -> 1678811811 Mar 14 17:36:51 r91-2 pacemaker-attrd[1629]: notice: Setting #digests-all[r91-1]: fence-scsi:fence_scsi:ec9ecba84b274fb35effbf2a47226087, -> fence-scsi:fence_scsi:41f4daa097914fe0b3f6ba8363f28cf9, Mar 14 17:36:51 r91-2 pacemaker-attrd[1629]: notice: Setting #digests-secure[r91-1]: fence-scsi:fence_scsi:8a7469d06699bb5cdf0da9304affaf6e, -> fence-scsi:fence_scsi:2467afb1330d3c0048abdc371dea6bc3, Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Transition 5 aborted by status-1-.node-unfenced doing modify #node-unfenced=1678811811: Transient attribute change Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Operation 'on' [2266] targeting r91-2 using fence-scsi returned 0 Mar 14 17:36:51 r91-2 pacemaker-fenced[1627]: notice: Operation 'on' targeting r91-2 by r91-2 for pacemaker-controld.1631@r91-2: OK (complete) Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Fence operation 9 for r91-2 passed Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: r91-2 was unfenced by r91-2 at the request of pacemaker-controld.1631@r91-2 Mar 14 17:36:51 r91-2 pacemaker-attrd[1629]: notice: Setting #node-unfenced[r91-2]: 1678811397 -> 1678811811 Mar 14 17:36:51 r91-2 pacemaker-attrd[1629]: notice: Setting #digests-all[r91-2]: fence-scsi:fence_scsi:ec9ecba84b274fb35effbf2a47226087, -> fence-scsi:fence_scsi:41f4daa097914fe0b3f6ba8363f28cf9, Mar 14 17:36:51 r91-2 pacemaker-attrd[1629]: notice: Setting #digests-secure[r91-2]: fence-scsi:fence_scsi:8a7469d06699bb5cdf0da9304affaf6e, -> fence-scsi:fence_scsi:2467afb1330d3c0048abdc371dea6bc3, Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Transition 5 (Complete=7, Pending=0, Fired=0, Skipped=3, Incomplete=4, Source=/var/lib/pacemaker/pengine/pe-input-205.bz2): Stopped Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: Actions: Start d1 ( r91-2 ) Mar 14 17:36:51 r91-2 pacemaker-schedulerd[1630]: notice: Calculated transition 6, saving inputs in /var/lib/pacemaker/pengine/pe-input-206.bz2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating monitor operation fence-scsi_monitor_60000 on r91-1 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating start operation d1_start_0 locally on r91-2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating monitor operation d2_monitor_10000 on r91-1 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Requesting local execution of start operation for d1 on r91-2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Result of start operation for d1 on r91-2: ok Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Initiating monitor operation d1_monitor_10000 locally on r91-2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Requesting local execution of monitor operation for d1 on r91-2 Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Result of monitor operation for d1 on r91-2: ok Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: Transition 6 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-206.bz2): Complete Mar 14 17:36:51 r91-2 pacemaker-controld[1631]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE With the last working version, see cib_before2.xml, cib_after2.xml: [root@r91-1 ~]# rpm -q pcs pacemaker pcs-0.11.3-4.el9_1.2.x86_64 pacemaker-2.1.4-2.el9.x86_64 [root@r91-1 ~]# journalctl -n0 -f Mar 14 17:53:30 r91-1 pacemaker-fenced[4320]: notice: Added 'fence-scsi' to device list (1 active device) [root@r91-2 ~]# journalctl -n0 -f Mar 14 17:53:30 r91-2 pacemaker-controld[4061]: notice: State transition S_IDLE -> S_POLICY_ENGINE Mar 14 17:53:30 r91-2 pacemaker-fenced[4057]: notice: Added 'fence-scsi' to device list (1 active device) Mar 14 17:53:30 r91-2 pacemaker-schedulerd[4060]: notice: Calculated transition 5, saving inputs in /var/lib/pacemaker/pengine/pe-input-215.bz2 Mar 14 17:53:30 r91-2 pacemaker-controld[4061]: notice: Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-215.bz2): Complete Mar 14 17:53:30 r91-2 pacemaker-controld[4061]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE --- Additional comment from Ken Gaillot on 2023-03-15 02:58:17 CET --- I need to investigate some more, but I think this is not a new problem, but an expansion of a case we neglected with the original bz. We're changing the digests in the resource history, but there is a separate copy of the digest stored in the "#digests-all" and "#digest-secure" node attributes, used to detect whether a node needs to be re-unfenced after a change in stonith device definition. Before 2.1.4-3, this only had an effect on Pacemaker Remote nodes, which is why we missed it. Since 2.1.4-3, it applies to all nodes. That was an important fix, so I think the only way around this will be for pcs to replace those node attribute values in addition to the digests in the resource history. The node attribute values look like: "stonith-fence_compute-fence-nova:fence_compute:ad312d85623cdb0a792e6fbd5e91a820," That is a comma-separated list of DEVICE_NAME:AGENT_NAME:DIGEST for each stonith device that has unfenced the node. "#digests-all" uses the all-parameter digest, and "#digest-secure" (which is only relevant to crm_simulate) uses the non-private-parameter digest. It's possible that's not the only issue, but that's definitely part of it.
Upstream commit: https://github.com/ClusterLabs/pcs/commit/bf7d33bdd41f6e51321ae66cd521cefc93acb3a4 Updated commands: * pcs stonith update-scsi-devices Test: Setup cluster with a shared storage and fence_scsi fencing. Setup enough resources in order to have each node running some resource. Use `pcs stonith update-scsi-devices` command to modify scsi devices of the fence_scsi stonith device. Check that resources did not restart (journalctl, crm_rsource --list-operations)
DevTestResults: [root@r08-09-a ~]# rpm -q pcs pcs-0.10.16-1.el8.x86_64 (pcs) [root@r08-09-a pcs]# pcs_test/suite --installed --traditional-verbose pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_1_nonrecurring_start_op_with_timeout (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_1_nonrecurring_start_op_with_timeout ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_1_monitor_with_timeout (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_1_monitor_with_timeout ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_2_monitor_ops_with_one_timeout (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_2_monitor_ops_with_one_timeout ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_no_monitor_ops (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_no_monitor_ops ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_default_monitor (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_default_monitor ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_2_monitor_ops_with_timeouts (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_2_monitor_ops_with_timeouts ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_digests_with_empty_value (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_digests_with_empty_value ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_all_nodes_multi_value (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_all_nodes_multi_value ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_all_nodes (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_all_nodes ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_not_on_all_nodes (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_not_on_all_nodes ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_not_all_digest_types (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_not_all_digest_types ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_no_digest_for_our_stonith_id (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_no_digest_for_our_stonith_id ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_without_last_comma (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_without_last_comma ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_without_digests_attrs (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_without_digests_attrs ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_not_on_all_nodes_multi_value (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_not_on_all_nodes_multi_value ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_various_start_ops_one_lrm_start_op (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_various_start_ops_one_lrm_start_op ... OK pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_without_last_comma_multi_value (subunit.RemotedTestCase) pcs_test.tier0.lib.commands.test_stonith_update_scsi_devices.TestUpdateScsiDevicesDigestsSetScsi.test_transient_digests_attrs_without_last_comma_multi_value ... OK ---------------------------------------------------------------------- Ran 17 tests in 0.096s OK Additional testing on real cluster by mlisik: root@r8-node-01 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.9 Beta (Ootpa) [root@r8-node-01 ~]# rpm -q pcs pacemaker pcs-0.10.16-1.el8.x86_64 pacemaker-2.1.6-1.el8.x86_64 [root@r8-node-01 ~]# export disk1=/dev/disk/by-id/scsi-3600140500e2fe60a3eb479bb39ca8d3d [root@r8-node-01 ~]# export disk2=/dev/disk/by-id/scsi-36001405fb15e3edf2994db380037abac [root@r8-node-01 ~]# export NODELIST=(r8-node-01 r8-node-02) [root@r8-node-01 ~]# pcs host auth -u hacluster -p password ${NODELIST[*]} r8-node-01: Authorized r8-node-02: Authorized [root@r8-node-01 ~]# pcs cluster setup HACluster ${NODELIST[*]} --start --wait No addresses specified for host 'r8-node-01', using 'r8-node-01' No addresses specified for host 'r8-node-02', using 'r8-node-02' Destroying cluster on hosts: 'r8-node-01', 'r8-node-02'... r8-node-01: Successfully destroyed cluster r8-node-02: Successfully destroyed cluster Requesting remove 'pcsd settings' from 'r8-node-01', 'r8-node-02' r8-node-01: successful removal of the file 'pcsd settings' r8-node-02: successful removal of the file 'pcsd settings' Sending 'corosync authkey', 'pacemaker authkey' to 'r8-node-01', 'r8-node-02' r8-node-01: successful distribution of the file 'corosync authkey' r8-node-01: successful distribution of the file 'pacemaker authkey' r8-node-02: successful distribution of the file 'corosync authkey' r8-node-02: successful distribution of the file 'pacemaker authkey' Sending 'corosync.conf' to 'r8-node-01', 'r8-node-02' r8-node-01: successful distribution of the file 'corosync.conf' r8-node-02: successful distribution of the file 'corosync.conf' Cluster has been successfully set up. Starting cluster on hosts: 'r8-node-01', 'r8-node-02'... Waiting for node(s) to start: 'r8-node-01', 'r8-node-02'... r8-node-01: Cluster started r8-node-02: Cluster started [root@r8-node-01 ~]# pcs stonith create fence-scsi fence_scsi devices=$disk1 pcmk_host_check=static-list pcmk_host_list="${NODELIST[*]}" pcmk_reboot_action=off meta provides=unfencing [root@r8-node-01 ~]# for i in $(seq 1 ${#NODELIST[@]}); do pcs resource create "d$i" ocf:pacemaker:Dummy; done [root@r8-node-01 ~]# pcs resource * d1 (ocf::pacemaker:Dummy): Started r8-node-02 * d2 (ocf::pacemaker:Dummy): Started r8-node-01 [root@r8-node-01 ~]# pcs stonith * fence-scsi (stonith:fence_scsi): Started r8-node-01 [root@r8-node-01 ~]# pcs stonith config Resource: fence-scsi (class=stonith type=fence_scsi) Attributes: fence-scsi-instance_attributes devices=/dev/disk/by-id/scsi-3600140500e2fe60a3eb479bb39ca8d3d pcmk_host_check=static-list pcmk_host_list="r8-node-01 r8-node-02" pcmk_reboot_action=off Meta Attributes: fence-scsi-meta_attributes provides=unfencing Operations: monitor: fence-scsi-monitor-interval-60s interval=60s [root@r8-node-01 ~]# for r in fence-scsi d1 d2; do crm_resource --resource $r --list-operations; done |& tee o1.txt fence-scsi (stonith:fence_scsi): Started: fence-scsi_start_0 (node=r8-node-01, call=6, rc=0, last-rc-change='Fri May 26 15:34:39 2023', exec=87ms): complete fence-scsi (stonith:fence_scsi): Started: fence-scsi_monitor_60000 (node=r8-node-01, call=7, rc=0, last-rc-change='Fri May 26 15:34:39 2023', exec=88ms): complete fence-scsi (stonith:fence_scsi): Started: fence-scsi_monitor_0 (node=r8-node-02, call=5, rc=7, last-rc-change='Fri May 26 15:34:39 2023', exec=2ms): complete d1 (ocf::pacemaker:Dummy): Started: d1_monitor_0 (node=r8-node-01, call=11, rc=7, last-rc-change='Fri May 26 15:34:40 2023', exec=14ms): complete d1 (ocf::pacemaker:Dummy): Started: d1_start_0 (node=r8-node-02, call=10, rc=0, last-rc-change='Fri May 26 15:34:40 2023', exec=18ms): complete d1 (ocf::pacemaker:Dummy): Started: d1_monitor_10000 (node=r8-node-02, call=11, rc=0, last-rc-change='Fri May 26 15:34:40 2023', exec=11ms): complete d2 (ocf::pacemaker:Dummy): Started: d2_start_0 (node=r8-node-01, call=16, rc=0, last-rc-change='Fri May 26 15:34:41 2023', exec=16ms): complete d2 (ocf::pacemaker:Dummy): Started: d2_monitor_10000 (node=r8-node-01, call=17, rc=0, last-rc-change='Fri May 26 15:34:41 2023', exec=13ms): complete d2 (ocf::pacemaker:Dummy): Started: d2_monitor_0 (node=r8-node-02, call=15, rc=7, last-rc-change='Fri May 26 15:34:41 2023', exec=20ms): complete [root@r8-node-01 ~]# pcs stonith update-scsi-devices fence-scsi add $disk2 [root@r8-node-01 ~]# for r in fence-scsi d1 d2; do crm_resource --resource $r --list-operations; done |& tee o2.txt fence-scsi (stonith:fence_scsi): Started: fence-scsi_start_0 (node=r8-node-01, call=6, rc=0, last-rc-change='Fri May 26 15:34:39 2023', exec=87ms): complete fence-scsi (stonith:fence_scsi): Started: fence-scsi_monitor_60000 (node=r8-node-01, call=7, rc=0, last-rc-change='Fri May 26 15:34:39 2023', exec=88ms): complete fence-scsi (stonith:fence_scsi): Started: fence-scsi_monitor_0 (node=r8-node-02, call=5, rc=7, last-rc-change='Fri May 26 15:34:39 2023', exec=2ms): complete d1 (ocf::pacemaker:Dummy): Started: d1_monitor_0 (node=r8-node-01, call=11, rc=7, last-rc-change='Fri May 26 15:34:40 2023', exec=14ms): complete d1 (ocf::pacemaker:Dummy): Started: d1_start_0 (node=r8-node-02, call=10, rc=0, last-rc-change='Fri May 26 15:34:40 2023', exec=18ms): complete d1 (ocf::pacemaker:Dummy): Started: d1_monitor_10000 (node=r8-node-02, call=11, rc=0, last-rc-change='Fri May 26 15:34:40 2023', exec=11ms): complete d2 (ocf::pacemaker:Dummy): Started: d2_start_0 (node=r8-node-01, call=16, rc=0, last-rc-change='Fri May 26 15:34:41 2023', exec=16ms): complete d2 (ocf::pacemaker:Dummy): Started: d2_monitor_10000 (node=r8-node-01, call=17, rc=0, last-rc-change='Fri May 26 15:34:41 2023', exec=13ms): complete d2 (ocf::pacemaker:Dummy): Started: d2_monitor_0 (node=r8-node-02, call=15, rc=7, last-rc-change='Fri May 26 15:34:41 2023', exec=20ms): complete [root@r8-node-01 ~]# diff -u o1.txt o2.txt [root@r8-node-01 ~]# echo $? 0 [root@r8-node-01 ~]# journalctl -n0 -f -- Logs begin at Fri 2023-05-26 15:09:03 CEST. -- May 26 15:39:47 r8-node-01 pacemaker-fenced[4373]: notice: Added 'fence-scsi' to device list (1 active device) [root@r8-node-02 ~]# journalctl -n0 -f -- Logs begin at Fri 2023-05-26 15:09:05 CEST. -- May 26 15:39:47 r8-node-02 pacemaker-controld[3703]: notice: State transition S_IDLE -> S_POLICY_ENGINE May 26 15:39:47 r8-node-02 pacemaker-fenced[3699]: notice: Added 'fence-scsi' to device list (1 active device) May 26 15:39:47 r8-node-02 pacemaker-schedulerd[3702]: notice: Calculated transition 5, saving inputs in /var/lib/pacemaker/pengine/pe-input-400.bz2 May 26 15:39:47 r8-node-02 pacemaker-controld[3703]: notice: Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-400.bz2): Complete May 26 15:39:47 r8-node-02 pacemaker-controld[3703]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [root@r8-node-01 ~]# pcs stonith config Resource: fence-scsi (class=stonith type=fence_scsi) Attributes: fence-scsi-instance_attributes devices=/dev/disk/by-id/scsi-3600140500e2fe60a3eb479bb39ca8d3d,/dev/disk/by- id/scsi-36001405fb15e3edf2994db380037abac pcmk_host_check=static-list pcmk_host_list="r8-node-01 r8-node-02" pcmk_reboot_action=off Meta Attributes: fence-scsi-meta_attributes provides=unfencing Operations: monitor: fence-scsi-monitor-interval-60s interval=60s RESULT: SCSI device was added to the stonith configuration and resources were not restarted.