2023845 – [RFE] Provide a way to add a scsi **mpath** fencing device to a cluster without requiring a restart of all cluster resources

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2023845 - [RFE] Provide a way to add a scsi **mpath** fencing device to a cluster without requiring a restart of all cluster resources

Summary: [RFE] Provide a way to add a scsi **mpath** fencing device to a cluster witho...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	pcs
Sub Component:
Version:	8.5
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	8.7
Assignee:	Miroslav Lisik
QA Contact:	cluster-qe@redhat.com
Docs Contact:	Steven J. Levine
URL:
Whiteboard:
Depends On:	1872376 1872378
Blocks:	2024522
TreeView+	depends on / blocked

Reported:	2021-11-16 16:55 UTC by Chris Feist
Modified:	2022-11-08 09:23 UTC (History)
CC List:	13 users (show)
Fixed In Version:	pcs-0.10.13-1.el8
Doc Type:	Enhancement
Doc Text:	.`pcs` command-line supports updating multipath SCSI devices without requiring a system restart You can now update multipath SCSI devices with the `pcs stonith update-scsi-devices` command. This command updates SCSI devices without causing a restart of other cluster resources running on the same node.
Clone Of:	1872378
Clones:	2024522 (view as bug list)
Environment:
Last Closed:	2022-11-08 09:12:53 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-102958	None	None	None	2021-11-16 16:59:29 UTC
Red Hat Knowledge Base (Solution)	4526971	None	None	None	2021-11-16 17:02:19 UTC
Red Hat Product Errata	RHSA-2022:7447	None	None	None	2022-11-08 09:13:11 UTC

Comment 5 Miroslav Lisik 2022-03-17 09:33:19 UTC

Upstream commit: https://github.com/ClusterLabs/pcs/commit/7eb7d3c2c57c6292c470c13af262afb9368f564b

Updated command:
pcs stonith update-scsi-devices

Test:
# pcs stonith update-scsi-devices <mpath-fence-device> set <device-path>...
# pcs stonith update-scsi-devices <mpath-fence-device> add <device-path>... remove <device-path>...

Devices of fence_mpath instance should be unfenced and updated and no resources should be restarted.

Comment 7 Miroslav Lisik 2022-05-26 08:49:21 UTC

DevTestResults:

[root@r8-node-01 ~]# rpm -q pcs
pcs-0.10.13-1.el8.x86_64

### shortcuts for disk paths

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do echo $d; done
/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb
/dev/disk/by-id/dm-uuid-mpath-36001405df3f8751c94649d3ae3f11d30
/dev/disk/by-id/dm-uuid-mpath-36001405eafb08de17a540bc9a97dedf3
/dev/disk/by-id/dm-uuid-mpath-360014054f2782eef9394e7aa72eea7e3

### multipath configuration

[root@r8-node-01 ~]# multipath -ll
mpatha (3600140526d1927596fd4577958a15dcb) dm-0 LIO-ORG,r8-mdisk-01
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:0 sdi 8:128 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 6:0:0:0  sda 8:0   active ready running
mpathb (36001405df3f8751c94649d3ae3f11d30) dm-1 LIO-ORG,r8-mdisk-02
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:1 sdl 8:176 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 6:0:0:1  sdd 8:48  active ready running
mpathc (36001405eafb08de17a540bc9a97dedf3) dm-2 LIO-ORG,r8-mdisk-03
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:2  sdc 8:32  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 10:0:0:2 sdk 8:160 active ready running
mpathd (360014054f2782eef9394e7aa72eea7e3) dm-3 LIO-ORG,r8-mdisk-04
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:3  sdb 8:16  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 10:0:0:3 sdj 8:144 active ready running

### list of reservation keys

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.

#### create fence-mpath stonith device and check the configuration

[root@r8-node-01 ~]# pcs stonith create mpath-fence-device fence_mpath devices="$mdisk1" pcmk_host_check="static-list" pcmk_host_list="r91-1 r91-2" pcmk_host_map="r8-node-01:1;r8-node-02:2;r8-node-03:3" pcmk_host_argument="key" pcmk_reboot_action="off"
[root@r8-node-01 ~]# pcs stonith config
Resource: mpath-fence-device (class=stonith type=fence_mpath)
  Attributes: mpath-fence-device-instance_attributes
    devices=/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb
    pcmk_host_argument=key
    pcmk_host_check=static-list
    pcmk_host_list="r91-1 r91-2"
    pcmk_host_map=r8-node-01:1;r8-node-02:2;r8-node-03:3
    pcmk_reboot_action=off
  Meta Attributes: mpath-fence-device-meta_attributes
    provides=unfencing
  Operations:
    monitor: mpath-fence-device-monitor-interval-60s
      interval=60s
[root@r8-node-01 ~]# pcs stonith
  * mpath-fence-device  (stonith:fence_mpath):   Started r8-node-02
[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.

### create some resources for restart detection checks

[root@r8-node-01 ~]# for i in $(seq -w 01 04); do pcs resource create d-$i ocf:pacemaker:Dummy; done
[root@r8-node-01 ~]# pcs resource
  * d-01        (ocf::pacemaker:Dummy):  Started r8-node-01
  * d-02        (ocf::pacemaker:Dummy):  Started r8-node-02
  * d-03        (ocf::pacemaker:Dummy):  Started r8-node-03
  * d-04        (ocf::pacemaker:Dummy):  Started r8-node-01


### update fence-mpath device configuration and check if the resources restarted

1) Set syntax

[root@r8-node-01 ~]# pcs stonith update-scsi-devices mpath-fence-device set $mdisk1 $mdisk2

[root@r8-node-01 ~]#
[root@r8-node-01 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:20 CEST. --
May 25 15:25:39 r8-node-01 pacemaker-fenced[97569]:  notice: Added 'mpath-fence-device' to device list (1 active device)

root@r8-node-02 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:23 CEST. --
May 25 15:25:38 r8-node-02 kernel: sd 6:0:0:1: reservation conflict
May 25 15:25:39 r8-node-02 pacemaker-fenced[88581]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-03 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:31 CEST. --
May 25 15:25:38 r8-node-03 kernel: sd 9:0:0:1: reservation conflict
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
May 25 15:25:39 r8-node-03 pacemaker-fenced[91327]:  notice: Added 'mpath-fence-device' to device list (1 active device)
May 25 15:25:39 r8-node-03 pacemaker-schedulerd[91330]:  notice: Calculated transition 28, saving inputs in /var/lib/pacemaker/pengine/pe-input-1560.bz2
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: Transition 28 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1560.bz2): Complete
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x1
    0x1
    0x3
    0x3
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.


2) Add/Remove syntax

[root@r8-node-01 ~]# pcs stonith update-scsi-devices mpath-fence-device add $mdisk3 $mdisk4 remove $mdisk2
[root@r8-node-01 ~]# pcs stonith config
Resource: mpath-fence-device (class=stonith type=fence_mpath)
  Attributes: mpath-fence-device-instance_attributes
    devices=/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb,/dev/disk/by-id/dm-uuid-mpath-360014054f2782eef9394e7aa72eea7e3,/dev/disk/by-id/dm-uuid-mpath-36001405eafb08de17a540bc9a97dedf3
    pcmk_host_argument=key
    pcmk_host_check=static-list
    pcmk_host_list="r91-1 r91-2"
    pcmk_host_map=r8-node-01:1;r8-node-02:2;r8-node-03:3
    pcmk_reboot_action=off
  Meta Attributes: mpath-fence-device-meta_attributes
    provides=unfencing
  Operations:
    monitor: mpath-fence-device-monitor-interval-60s
      interval=60s

[root@r8-node-01 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:20 CEST. --
May 25 16:46:21 r8-node-01 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 10:0:0:3: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 10:0:0:2: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-01 pacemaker-fenced[97569]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-02 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:23 CEST. --
May 25 16:46:21 r8-node-02 kernel: sd 7:0:0:3: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 7:0:0:2: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-02 pacemaker-fenced[88581]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-03 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:31 CEST. --
May 25 16:46:21 r8-node-03 kernel: sd 9:0:0:3: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 9:0:0:2: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
May 25 16:46:22 r8-node-03 pacemaker-fenced[91327]:  notice: Added 'mpath-fence-device' to device list (1 active device)
May 25 16:46:22 r8-node-03 pacemaker-schedulerd[91330]:  notice: Calculated transition 35, saving inputs in /var/lib/pacemaker/pengine/pe-input-1563.bz2
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: Transition 35 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1563.bz2): Complete
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x1
    0x1
    0x3
    0x3
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x3
    0x3
    0x1
    0x1
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x3
    0x3
    0x1
    0x1

Comment 19 errata-xmlrpc 2022-11-08 09:12:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: pcs security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7447

Note You need to log in before you can comment on or make changes to this bug.