Bug 2023845 - [RFE] Provide a way to add a scsi **mpath** fencing device to a cluster without requiring a restart of all cluster resources
Summary: [RFE] Provide a way to add a scsi **mpath** fencing device to a cluster witho...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: pcs
Version: 8.5
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 8.7
Assignee: Miroslav Lisik
QA Contact: cluster-qe@redhat.com
Steven J. Levine
URL:
Whiteboard:
Depends On: 1872376 1872378
Blocks: 2024522
TreeView+ depends on / blocked
 
Reported: 2021-11-16 16:55 UTC by Chris Feist
Modified: 2022-11-08 09:23 UTC (History)
13 users (show)

Fixed In Version: pcs-0.10.13-1.el8
Doc Type: Enhancement
Doc Text:
.`pcs` command-line supports updating multipath SCSI devices without requiring a system restart You can now update multipath SCSI devices with the `pcs stonith update-scsi-devices` command. This command updates SCSI devices without causing a restart of other cluster resources running on the same node.
Clone Of: 1872378
: 2024522 (view as bug list)
Environment:
Last Closed: 2022-11-08 09:12:53 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-102958 0 None None None 2021-11-16 16:59:29 UTC
Red Hat Knowledge Base (Solution) 4526971 0 None None None 2021-11-16 17:02:19 UTC
Red Hat Product Errata RHSA-2022:7447 0 None None None 2022-11-08 09:13:11 UTC

Comment 5 Miroslav Lisik 2022-03-17 09:33:19 UTC
Upstream commit: https://github.com/ClusterLabs/pcs/commit/7eb7d3c2c57c6292c470c13af262afb9368f564b

Updated command:
pcs stonith update-scsi-devices

Test:
# pcs stonith update-scsi-devices <mpath-fence-device> set <device-path>...
# pcs stonith update-scsi-devices <mpath-fence-device> add <device-path>... remove <device-path>...

Devices of fence_mpath instance should be unfenced and updated and no resources should be restarted.

Comment 7 Miroslav Lisik 2022-05-26 08:49:21 UTC
DevTestResults:

[root@r8-node-01 ~]# rpm -q pcs
pcs-0.10.13-1.el8.x86_64

### shortcuts for disk paths

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do echo $d; done
/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb
/dev/disk/by-id/dm-uuid-mpath-36001405df3f8751c94649d3ae3f11d30
/dev/disk/by-id/dm-uuid-mpath-36001405eafb08de17a540bc9a97dedf3
/dev/disk/by-id/dm-uuid-mpath-360014054f2782eef9394e7aa72eea7e3

### multipath configuration

[root@r8-node-01 ~]# multipath -ll
mpatha (3600140526d1927596fd4577958a15dcb) dm-0 LIO-ORG,r8-mdisk-01
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:0 sdi 8:128 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 6:0:0:0  sda 8:0   active ready running
mpathb (36001405df3f8751c94649d3ae3f11d30) dm-1 LIO-ORG,r8-mdisk-02
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:1 sdl 8:176 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 6:0:0:1  sdd 8:48  active ready running
mpathc (36001405eafb08de17a540bc9a97dedf3) dm-2 LIO-ORG,r8-mdisk-03
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:2  sdc 8:32  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 10:0:0:2 sdk 8:160 active ready running
mpathd (360014054f2782eef9394e7aa72eea7e3) dm-3 LIO-ORG,r8-mdisk-04
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:3  sdb 8:16  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 10:0:0:3 sdj 8:144 active ready running

### list of reservation keys

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.

#### create fence-mpath stonith device and check the configuration

[root@r8-node-01 ~]# pcs stonith create mpath-fence-device fence_mpath devices="$mdisk1" pcmk_host_check="static-list" pcmk_host_list="r91-1 r91-2" pcmk_host_map="r8-node-01:1;r8-node-02:2;r8-node-03:3" pcmk_host_argument="key" pcmk_reboot_action="off"
[root@r8-node-01 ~]# pcs stonith config
Resource: mpath-fence-device (class=stonith type=fence_mpath)
  Attributes: mpath-fence-device-instance_attributes
    devices=/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb
    pcmk_host_argument=key
    pcmk_host_check=static-list
    pcmk_host_list="r91-1 r91-2"
    pcmk_host_map=r8-node-01:1;r8-node-02:2;r8-node-03:3
    pcmk_reboot_action=off
  Meta Attributes: mpath-fence-device-meta_attributes
    provides=unfencing
  Operations:
    monitor: mpath-fence-device-monitor-interval-60s
      interval=60s
[root@r8-node-01 ~]# pcs stonith
  * mpath-fence-device  (stonith:fence_mpath):   Started r8-node-02
[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.

### create some resources for restart detection checks

[root@r8-node-01 ~]# for i in $(seq -w 01 04); do pcs resource create d-$i ocf:pacemaker:Dummy; done
[root@r8-node-01 ~]# pcs resource
  * d-01        (ocf::pacemaker:Dummy):  Started r8-node-01
  * d-02        (ocf::pacemaker:Dummy):  Started r8-node-02
  * d-03        (ocf::pacemaker:Dummy):  Started r8-node-03
  * d-04        (ocf::pacemaker:Dummy):  Started r8-node-01


### update fence-mpath device configuration and check if the resources restarted

1) Set syntax

[root@r8-node-01 ~]# pcs stonith update-scsi-devices mpath-fence-device set $mdisk1 $mdisk2

[root@r8-node-01 ~]#
[root@r8-node-01 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:20 CEST. --
May 25 15:25:39 r8-node-01 pacemaker-fenced[97569]:  notice: Added 'mpath-fence-device' to device list (1 active device)

root@r8-node-02 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:23 CEST. --
May 25 15:25:38 r8-node-02 kernel: sd 6:0:0:1: reservation conflict
May 25 15:25:39 r8-node-02 pacemaker-fenced[88581]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-03 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:31 CEST. --
May 25 15:25:38 r8-node-03 kernel: sd 9:0:0:1: reservation conflict
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
May 25 15:25:39 r8-node-03 pacemaker-fenced[91327]:  notice: Added 'mpath-fence-device' to device list (1 active device)
May 25 15:25:39 r8-node-03 pacemaker-schedulerd[91330]:  notice: Calculated transition 28, saving inputs in /var/lib/pacemaker/pengine/pe-input-1560.bz2
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: Transition 28 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1560.bz2): Complete
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x1
    0x1
    0x3
    0x3
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.


2) Add/Remove syntax

[root@r8-node-01 ~]# pcs stonith update-scsi-devices mpath-fence-device add $mdisk3 $mdisk4 remove $mdisk2
[root@r8-node-01 ~]# pcs stonith config
Resource: mpath-fence-device (class=stonith type=fence_mpath)
  Attributes: mpath-fence-device-instance_attributes
    devices=/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb,/dev/disk/by-id/dm-uuid-mpath-360014054f2782eef9394e7aa72eea7e3,/dev/disk/by-id/dm-uuid-mpath-36001405eafb08de17a540bc9a97dedf3
    pcmk_host_argument=key
    pcmk_host_check=static-list
    pcmk_host_list="r91-1 r91-2"
    pcmk_host_map=r8-node-01:1;r8-node-02:2;r8-node-03:3
    pcmk_reboot_action=off
  Meta Attributes: mpath-fence-device-meta_attributes
    provides=unfencing
  Operations:
    monitor: mpath-fence-device-monitor-interval-60s
      interval=60s

[root@r8-node-01 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:20 CEST. --
May 25 16:46:21 r8-node-01 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 10:0:0:3: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 10:0:0:2: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-01 pacemaker-fenced[97569]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-02 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:23 CEST. --
May 25 16:46:21 r8-node-02 kernel: sd 7:0:0:3: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 7:0:0:2: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-02 pacemaker-fenced[88581]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-03 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:31 CEST. --
May 25 16:46:21 r8-node-03 kernel: sd 9:0:0:3: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 9:0:0:2: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
May 25 16:46:22 r8-node-03 pacemaker-fenced[91327]:  notice: Added 'mpath-fence-device' to device list (1 active device)
May 25 16:46:22 r8-node-03 pacemaker-schedulerd[91330]:  notice: Calculated transition 35, saving inputs in /var/lib/pacemaker/pengine/pe-input-1563.bz2
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: Transition 35 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1563.bz2): Complete
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x1
    0x1
    0x3
    0x3
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x3
    0x3
    0x1
    0x1
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x3
    0x3
    0x1
    0x1

Comment 19 errata-xmlrpc 2022-11-08 09:12:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: pcs security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7447


Note You need to log in before you can comment on or make changes to this bug.